[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:04:53.028291 22020 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.129.62:34865
I20240502 14:04:53.029515 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:53.030539 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:53.046552 22026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:53.047120 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:53.047516 22029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:53.047937 22020 server_base.cc:1034] running on GCE node
I20240502 14:04:54.095582 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:54.095819 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:54.095968 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658694095952 us; error 0 us; skew 500 ppm
I20240502 14:04:54.096745 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:54.102663 22020 webserver.cc:460] Webserver started at http://127.21.129.62:39741/ using document root <none> and password file <none>
I20240502 14:04:54.103504 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:54.103698 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:54.104056 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:54.110023 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/master-0-root/instance:
uuid: "6f4c5170f2004240ba6da1c988297f20"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.117192 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.009s
I20240502 14:04:54.121982 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.123049 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:04:54.123404 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/master-0-root
uuid: "6f4c5170f2004240ba6da1c988297f20"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.123734 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:54.184360 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:54.185688 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:54.186084 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:54.250098 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.62:34865
I20240502 14:04:54.250209 22087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.62:34865 every 8 connection(s)
I20240502 14:04:54.256603 22088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:54.278926 22088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20: Bootstrap starting.
I20240502 14:04:54.284206 22088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:54.285898 22088 log.cc:824] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:54.289909 22088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20: No bootstrap required, opened a new log
I20240502 14:04:54.313304 22088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4c5170f2004240ba6da1c988297f20" member_type: VOTER }
I20240502 14:04:54.313807 22088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:54.314200 22088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4c5170f2004240ba6da1c988297f20, State: Initialized, Role: FOLLOWER
I20240502 14:04:54.314998 22088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4c5170f2004240ba6da1c988297f20" member_type: VOTER }
I20240502 14:04:54.315490 22088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:54.315722 22088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:54.316009 22088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:54.324514 22088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4c5170f2004240ba6da1c988297f20" member_type: VOTER }
I20240502 14:04:54.325243 22088 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f4c5170f2004240ba6da1c988297f20; no voters:
I20240502 14:04:54.326884 22088 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:54.327243 22091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:54.329761 22091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4c5170f2004240ba6da1c988297f20, State: Running, Role: LEADER
I20240502 14:04:54.330618 22091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4c5170f2004240ba6da1c988297f20" member_type: VOTER }
I20240502 14:04:54.331650 22088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:54.341122 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f4c5170f2004240ba6da1c988297f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4c5170f2004240ba6da1c988297f20" member_type: VOTER } }
I20240502 14:04:54.341933 22093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4c5170f2004240ba6da1c988297f20. Latest consensus state: current_term: 1 leader_uuid: "6f4c5170f2004240ba6da1c988297f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4c5170f2004240ba6da1c988297f20" member_type: VOTER } }
I20240502 14:04:54.342593 22093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:54.343643 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:54.346215 22101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:54.356743 22101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:54.360543 22020 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:04:54.373531 22101 catalog_manager.cc:1347] Generated new cluster ID: b8d1bded4bbd4f3492670a3cadea51a6
I20240502 14:04:54.373740 22101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:54.396395 22101 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:54.397677 22101 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:54.410151 22101 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20: Generated new TSK 0
I20240502 14:04:54.410871 22101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:04:54.427009 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:54.436235 22110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:54.437187 22111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:54.440608 22020 server_base.cc:1034] running on GCE node
W20240502 14:04:54.441198 22113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:54.441927 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:54.442116 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:54.442293 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658694442275 us; error 0 us; skew 500 ppm
I20240502 14:04:54.443038 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:54.447433 22020 webserver.cc:460] Webserver started at http://127.21.129.1:41027/ using document root <none> and password file <none>
I20240502 14:04:54.447954 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:54.448144 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:54.448395 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:54.449431 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-0-root/instance:
uuid: "eef3139068244521a464f0100bc1d29c"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.455001 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20240502 14:04:54.458900 22118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.459823 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:04:54.460168 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-0-root
uuid: "eef3139068244521a464f0100bc1d29c"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.460529 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:54.505378 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:54.506948 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:54.509645 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:54.514825 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:54.515129 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.515457 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:54.515683 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.600756 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.1:43373
I20240502 14:04:54.600823 22180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.1:43373 every 8 connection(s)
I20240502 14:04:54.609158 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:54.618294 22185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:54.622146 22186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:54.625995 22188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:54.629711 22020 server_base.cc:1034] running on GCE node
I20240502 14:04:54.630443 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:54.630650 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:54.630836 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658694630804 us; error 0 us; skew 500 ppm
I20240502 14:04:54.631369 22181 heartbeater.cc:347] Connected to a master server at 127.21.129.62:34865
I20240502 14:04:54.631575 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:54.631857 22181 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:54.633280 22181 heartbeater.cc:511] Master 127.21.129.62:34865 requested a full tablet report, sending...
I20240502 14:04:54.635141 22020 webserver.cc:460] Webserver started at http://127.21.129.2:34729/ using document root <none> and password file <none>
I20240502 14:04:54.635726 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:54.635918 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:54.636195 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:54.636595 22052 ts_manager.cc:194] Registered new tserver with Master: eef3139068244521a464f0100bc1d29c (127.21.129.1:43373)
I20240502 14:04:54.637288 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-1-root/instance:
uuid: "8c3a0e50105649c3b949b797345e463e"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.639444 22052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51192
I20240502 14:04:54.643816 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 14:04:54.647764 22193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.648574 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 14:04:54.648882 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-1-root
uuid: "8c3a0e50105649c3b949b797345e463e"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.649199 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:54.678282 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:54.679463 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:54.681380 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:54.684137 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:54.684320 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.684525 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:54.684677 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.767724 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.2:37501
I20240502 14:04:54.767908 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.2:37501 every 8 connection(s)
I20240502 14:04:54.773677 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:54.782138 22260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:54.785245 22261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:54.787763 22263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:54.789384 22020 server_base.cc:1034] running on GCE node
I20240502 14:04:54.790210 22257 heartbeater.cc:347] Connected to a master server at 127.21.129.62:34865
I20240502 14:04:54.790383 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:04:54.790594 22257 heartbeater.cc:464] Registering TS with master...
W20240502 14:04:54.790630 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:54.790927 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658694790909 us; error 0 us; skew 500 ppm
I20240502 14:04:54.791563 22257 heartbeater.cc:511] Master 127.21.129.62:34865 requested a full tablet report, sending...
I20240502 14:04:54.791730 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:54.793694 22052 ts_manager.cc:194] Registered new tserver with Master: 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:54.794435 22020 webserver.cc:460] Webserver started at http://127.21.129.3:44549/ using document root <none> and password file <none>
I20240502 14:04:54.794881 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:54.795069 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:54.795321 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:54.795384 22052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51208
I20240502 14:04:54.796564 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-2-root/instance:
uuid: "6151fcb16d6d4d01b2ea65767bda6076"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.801086 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:04:54.804175 22268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.804961 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:04:54.805294 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-2-root
uuid: "6151fcb16d6d4d01b2ea65767bda6076"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-bdl7"
I20240502 14:04:54.805660 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714658692869395-22020-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:54.818270 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:54.819826 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:54.822230 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:54.825842 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:54.826113 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.826390 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:54.826610 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.909881 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.3:43421
I20240502 14:04:54.909955 22330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.3:43421 every 8 connection(s)
I20240502 14:04:54.924674 22331 heartbeater.cc:347] Connected to a master server at 127.21.129.62:34865
I20240502 14:04:54.925004 22331 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:54.925669 22331 heartbeater.cc:511] Master 127.21.129.62:34865 requested a full tablet report, sending...
I20240502 14:04:54.927515 22052 ts_manager.cc:194] Registered new tserver with Master: 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:54.927979 22020 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013179374s
I20240502 14:04:54.929251 22052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51214
I20240502 14:04:54.932426 22020 test_util.cc:268] Using random seed: 376186804
I20240502 14:04:54.979434 22052 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51222:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:54.982954 22052 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:55.083783 22221 tablet_service.cc:1467] Processing CreateTablet for tablet 3e656cba3bd84eb893113b23c825e654 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 14:04:55.085016 22221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e656cba3bd84eb893113b23c825e654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.087594 22222 tablet_service.cc:1467] Processing CreateTablet for tablet e786d30859f04ad89ac47caa0cb53622 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 14:04:55.089514 22220 tablet_service.cc:1467] Processing CreateTablet for tablet 848dc17f220448abb94b378fb2999425 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 14:04:55.090011 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e786d30859f04ad89ac47caa0cb53622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.094177 22220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848dc17f220448abb94b378fb2999425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.092852 22219 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7d83ca7ba84da093d1d973c934eb95 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 14:04:55.096210 22219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7d83ca7ba84da093d1d973c934eb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.108695 22218 tablet_service.cc:1467] Processing CreateTablet for tablet 669279922bc24f55908b209483dd8271 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 14:04:55.109884 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669279922bc24f55908b209483dd8271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.114521 22146 tablet_service.cc:1467] Processing CreateTablet for tablet e786d30859f04ad89ac47caa0cb53622 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 14:04:55.115788 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e786d30859f04ad89ac47caa0cb53622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.116802 22217 tablet_service.cc:1467] Processing CreateTablet for tablet d091765cb37b415ba7ebbde7e534576f (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 14:04:55.119347 22217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d091765cb37b415ba7ebbde7e534576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.128938 22145 tablet_service.cc:1467] Processing CreateTablet for tablet 3e656cba3bd84eb893113b23c825e654 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 14:04:55.128649 22215 tablet_service.cc:1467] Processing CreateTablet for tablet 491e9d03d7da4dd3a9ab09767574a167 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 14:04:55.130159 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e656cba3bd84eb893113b23c825e654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.131654 22215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491e9d03d7da4dd3a9ab09767574a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.145395 22144 tablet_service.cc:1467] Processing CreateTablet for tablet 848dc17f220448abb94b378fb2999425 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 14:04:55.150823 22214 tablet_service.cc:1467] Processing CreateTablet for tablet 172d1f7d66f54633a1423f26d55ba409 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 14:04:55.149096 22213 tablet_service.cc:1467] Processing CreateTablet for tablet 0f546cfa74964958816fa7b370229792 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 14:04:55.161706 22214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172d1f7d66f54633a1423f26d55ba409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.163555 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848dc17f220448abb94b378fb2999425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.158704 22142 tablet_service.cc:1467] Processing CreateTablet for tablet 669279922bc24f55908b209483dd8271 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 14:04:55.168207 22139 tablet_service.cc:1467] Processing CreateTablet for tablet 172d1f7d66f54633a1423f26d55ba409 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 14:04:55.169021 22138 tablet_service.cc:1467] Processing CreateTablet for tablet 0f546cfa74964958816fa7b370229792 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 14:04:55.171108 22213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f546cfa74964958816fa7b370229792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.161751 22141 tablet_service.cc:1467] Processing CreateTablet for tablet d091765cb37b415ba7ebbde7e534576f (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 14:04:55.161110 22143 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7d83ca7ba84da093d1d973c934eb95 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 14:04:55.175091 22140 tablet_service.cc:1467] Processing CreateTablet for tablet 491e9d03d7da4dd3a9ab09767574a167 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 14:04:55.179705 22349 tablet_bootstrap.cc:492] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.173637 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669279922bc24f55908b209483dd8271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.192258 22141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d091765cb37b415ba7ebbde7e534576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.195677 22139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172d1f7d66f54633a1423f26d55ba409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.203510 22349 tablet_bootstrap.cc:654] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.213884 22143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7d83ca7ba84da093d1d973c934eb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.220541 22350 tablet_bootstrap.cc:492] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.223436 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7d83ca7ba84da093d1d973c934eb95 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 14:04:55.226053 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 848dc17f220448abb94b378fb2999425 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 14:04:55.226773 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 669279922bc24f55908b209483dd8271 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 14:04:55.227490 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848dc17f220448abb94b378fb2999425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.233387 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491e9d03d7da4dd3a9ab09767574a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.218302 22296 tablet_service.cc:1467] Processing CreateTablet for tablet e786d30859f04ad89ac47caa0cb53622 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 14:04:55.231998 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 3e656cba3bd84eb893113b23c825e654 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 14:04:55.237633 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669279922bc24f55908b209483dd8271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.243731 22291 tablet_service.cc:1467] Processing CreateTablet for tablet d091765cb37b415ba7ebbde7e534576f (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 14:04:55.248797 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f546cfa74964958816fa7b370229792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.252442 22290 tablet_service.cc:1467] Processing CreateTablet for tablet 491e9d03d7da4dd3a9ab09767574a167 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 14:04:55.256543 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7d83ca7ba84da093d1d973c934eb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.249393 22289 tablet_service.cc:1467] Processing CreateTablet for tablet 172d1f7d66f54633a1423f26d55ba409 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 14:04:55.255893 22288 tablet_service.cc:1467] Processing CreateTablet for tablet 0f546cfa74964958816fa7b370229792 (DEFAULT_TABLE table=test-workload [id=7b9df96901ed49f1bb71c728a1dbbfed]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 14:04:55.270246 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172d1f7d66f54633a1423f26d55ba409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.278041 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e786d30859f04ad89ac47caa0cb53622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.293723 22349 tablet_bootstrap.cc:492] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.294227 22349 ts_tablet_manager.cc:1367] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.129s user 0.024s sys 0.026s
I20240502 14:04:55.295648 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e656cba3bd84eb893113b23c825e654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.297732 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d091765cb37b415ba7ebbde7e534576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.301930 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f546cfa74964958816fa7b370229792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.305987 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491e9d03d7da4dd3a9ab09767574a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.326267 22349 raft_consensus.cc:348] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.327039 22349 raft_consensus.cc:374] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.327365 22349 raft_consensus.cc:729] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.328135 22349 consensus_queue.cc:260] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.345256 22353 tablet_bootstrap.cc:492] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
W20240502 14:04:55.351025 22100 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:04:55.358548 22257 heartbeater.cc:503] Master 127.21.129.62:34865 was elected leader, sending a full tablet report...
I20240502 14:04:55.367323 22350 tablet_bootstrap.cc:654] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.380632 22349 ts_tablet_manager.cc:1398] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.086s user 0.011s sys 0.008s
I20240502 14:04:55.409076 22349 tablet_bootstrap.cc:492] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.419067 22353 tablet_bootstrap.cc:654] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.423457 22349 tablet_bootstrap.cc:654] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.435773 22350 tablet_bootstrap.cc:492] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.436336 22350 ts_tablet_manager.cc:1367] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.242s user 0.035s sys 0.025s
I20240502 14:04:55.438928 22350 raft_consensus.cc:348] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.445004 22350 raft_consensus.cc:374] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.445282 22350 raft_consensus.cc:729] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.445962 22350 consensus_queue.cc:260] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.463043 22350 ts_tablet_manager.cc:1398] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.026s user 0.006s sys 0.003s
I20240502 14:04:55.463572 22353 tablet_bootstrap.cc:492] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.463873 22353 ts_tablet_manager.cc:1367] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.119s user 0.022s sys 0.009s
I20240502 14:04:55.464208 22349 tablet_bootstrap.cc:492] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.464639 22349 ts_tablet_manager.cc:1367] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.007s
I20240502 14:04:55.465641 22353 raft_consensus.cc:348] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.466111 22353 raft_consensus.cc:374] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.466295 22353 raft_consensus.cc:729] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.466995 22181 heartbeater.cc:503] Master 127.21.129.62:34865 was elected leader, sending a full tablet report...
I20240502 14:04:55.466795 22353 consensus_queue.cc:260] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.467372 22350 tablet_bootstrap.cc:492] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.470355 22349 raft_consensus.cc:348] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.471046 22349 raft_consensus.cc:374] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.471349 22349 raft_consensus.cc:729] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.471971 22349 consensus_queue.cc:260] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.473131 22350 tablet_bootstrap.cc:654] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.473960 22349 ts_tablet_manager.cc:1398] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:04:55.474773 22349 tablet_bootstrap.cc:492] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.478637 22331 heartbeater.cc:503] Master 127.21.129.62:34865 was elected leader, sending a full tablet report...
I20240502 14:04:55.479597 22353 ts_tablet_manager.cc:1398] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20240502 14:04:55.480216 22349 tablet_bootstrap.cc:654] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.480381 22353 tablet_bootstrap.cc:492] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.486100 22353 tablet_bootstrap.cc:654] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.492035 22350 tablet_bootstrap.cc:492] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.492486 22350 ts_tablet_manager.cc:1367] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.006s
I20240502 14:04:55.495013 22350 raft_consensus.cc:348] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.495738 22350 raft_consensus.cc:374] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.496029 22350 raft_consensus.cc:729] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.496814 22350 consensus_queue.cc:260] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.497678 22353 tablet_bootstrap.cc:492] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.498085 22353 ts_tablet_manager.cc:1367] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20240502 14:04:55.498978 22350 ts_tablet_manager.cc:1398] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:04:55.499792 22350 tablet_bootstrap.cc:492] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.500530 22353 raft_consensus.cc:348] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.501222 22353 raft_consensus.cc:374] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.501504 22353 raft_consensus.cc:729] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.502625 22353 consensus_queue.cc:260] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.505331 22353 ts_tablet_manager.cc:1398] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 14:04:55.506175 22353 tablet_bootstrap.cc:492] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.506448 22350 tablet_bootstrap.cc:654] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.518124 22350 tablet_bootstrap.cc:492] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.518517 22350 ts_tablet_manager.cc:1367] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20240502 14:04:55.519502 22353 tablet_bootstrap.cc:654] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.520936 22350 raft_consensus.cc:348] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.521646 22350 raft_consensus.cc:374] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.521939 22350 raft_consensus.cc:729] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.522656 22350 consensus_queue.cc:260] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
W20240502 14:04:55.524475 22258 tablet.cc:2369] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:04:55.524843 22350 ts_tablet_manager.cc:1398] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 14:04:55.525772 22350 tablet_bootstrap.cc:492] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.531426 22350 tablet_bootstrap.cc:654] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.533653 22349 tablet_bootstrap.cc:492] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.534072 22349 ts_tablet_manager.cc:1367] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.027s
I20240502 14:04:55.536878 22349 raft_consensus.cc:348] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.537391 22353 tablet_bootstrap.cc:492] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.537700 22349 raft_consensus.cc:374] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.537879 22353 ts_tablet_manager.cc:1367] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.010s
I20240502 14:04:55.538252 22349 raft_consensus.cc:729] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.539103 22349 consensus_queue.cc:260] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.540692 22353 raft_consensus.cc:348] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.541425 22353 raft_consensus.cc:374] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.541708 22353 raft_consensus.cc:729] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.543735 22349 ts_tablet_manager.cc:1398] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 14:04:55.544651 22349 tablet_bootstrap.cc:492] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.545209 22353 consensus_queue.cc:260] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.549238 22353 ts_tablet_manager.cc:1398] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.011s user 0.000s sys 0.005s
I20240502 14:04:55.550160 22350 tablet_bootstrap.cc:492] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.550580 22350 ts_tablet_manager.cc:1367] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20240502 14:04:55.552107 22357 raft_consensus.cc:482] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.552578 22357 raft_consensus.cc:504] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.553042 22350 raft_consensus.cc:348] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.553751 22350 raft_consensus.cc:374] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.554040 22350 raft_consensus.cc:729] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.554647 22353 tablet_bootstrap.cc:492] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.554637 22350 consensus_queue.cc:260] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.555946 22357 leader_election.cc:288] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:55.574044 22353 tablet_bootstrap.cc:654] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.577174 22350 ts_tablet_manager.cc:1398] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.026s user 0.016s sys 0.004s
I20240502 14:04:55.577786 22349 tablet_bootstrap.cc:654] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.578109 22350 tablet_bootstrap.cc:492] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.585960 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:55.586722 22232 raft_consensus.cc:2454] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 0.
I20240502 14:04:55.588075 22120 leader_election.cc:302] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:55.589022 22366 raft_consensus.cc:2781] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.589445 22366 raft_consensus.cc:482] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.589773 22366 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.593462 22350 tablet_bootstrap.cc:654] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.596493 22366 raft_consensus.cc:504] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.598486 22366 leader_election.cc:288] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:55.599112 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:55.599675 22232 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.602191 22353 tablet_bootstrap.cc:492] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.602622 22353 ts_tablet_manager.cc:1367] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.001s
I20240502 14:04:55.603586 22349 tablet_bootstrap.cc:492] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.604146 22349 ts_tablet_manager.cc:1367] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.006s
I20240502 14:04:55.604753 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:55.605243 22353 raft_consensus.cc:348] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.605957 22353 raft_consensus.cc:374] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.606314 22353 raft_consensus.cc:729] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
W20240502 14:04:55.606617 22122 leader_election.cc:341] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:55.603742 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:55.606966 22353 consensus_queue.cc:260] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.607306 22349 raft_consensus.cc:348] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.608366 22349 raft_consensus.cc:374] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.608670 22349 raft_consensus.cc:729] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
W20240502 14:04:55.609998 22122 leader_election.cc:341] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:55.612115 22350 tablet_bootstrap.cc:492] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.612581 22350 ts_tablet_manager.cc:1367] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.012s
I20240502 14:04:55.612965 22232 raft_consensus.cc:2454] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 1.
I20240502 14:04:55.609400 22353 ts_tablet_manager.cc:1398] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20240502 14:04:55.613976 22120 leader_election.cc:302] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters: 6151fcb16d6d4d01b2ea65767bda6076
I20240502 14:04:55.615274 22350 raft_consensus.cc:348] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.615798 22366 raft_consensus.cc:2781] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.615868 22350 raft_consensus.cc:374] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.616343 22366 raft_consensus.cc:686] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:55.616350 22350 raft_consensus.cc:729] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.617175 22366 consensus_queue.cc:237] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.617625 22350 consensus_queue.cc:260] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.619969 22350 ts_tablet_manager.cc:1398] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:04:55.620616 22350 tablet_bootstrap.cc:492] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.621116 22353 tablet_bootstrap.cc:492] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.609378 22349 consensus_queue.cc:260] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.625069 22350 tablet_bootstrap.cc:654] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.626812 22349 ts_tablet_manager.cc:1398] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240502 14:04:55.629087 22349 tablet_bootstrap.cc:492] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.636756 22352 raft_consensus.cc:482] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.637199 22352 raft_consensus.cc:504] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.640110 22352 leader_election.cc:288] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:55.641222 22349 tablet_bootstrap.cc:654] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.645375 22052 catalog_manager.cc:5484] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 0 to 1, leader changed from <none> to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 1 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:55.649300 22358 raft_consensus.cc:482] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.649784 22358 raft_consensus.cc:504] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.652202 22358 leader_election.cc:288] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.665592 22349 tablet_bootstrap.cc:492] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.667871 22349 ts_tablet_manager.cc:1367] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.015s
I20240502 14:04:55.680549 22349 raft_consensus.cc:348] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.685048 22349 raft_consensus.cc:374] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.674098 22353 tablet_bootstrap.cc:654] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.688369 22366 raft_consensus.cc:482] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.688858 22366 raft_consensus.cc:504] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.690786 22366 leader_election.cc:288] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:55.691824 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:55.692487 22306 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 0.
I20240502 14:04:55.692654 22349 raft_consensus.cc:729] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.693625 22122 leader_election.cc:302] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:55.694396 22366 raft_consensus.cc:2781] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.694744 22366 raft_consensus.cc:482] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.695055 22366 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.695140 22353 tablet_bootstrap.cc:492] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.697065 22353 ts_tablet_manager.cc:1367] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.000s
I20240502 14:04:55.699646 22353 raft_consensus.cc:348] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.700327 22353 raft_consensus.cc:374] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.700610 22353 raft_consensus.cc:729] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.701261 22353 consensus_queue.cc:260] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.701573 22366 raft_consensus.cc:504] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.709964 22353 ts_tablet_manager.cc:1398] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20240502 14:04:55.710350 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:55.710886 22353 tablet_bootstrap.cc:492] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.711872 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:55.716466 22366 leader_election.cc:288] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:55.716670 22353 tablet_bootstrap.cc:654] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.717504 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:55.718076 22306 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.693933 22349 consensus_queue.cc:260] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.723346 22232 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 0.
I20240502 14:04:55.724038 22349 ts_tablet_manager.cc:1398] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.046s user 0.006s sys 0.001s
I20240502 14:04:55.725100 22349 tablet_bootstrap.cc:492] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.727072 22306 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 1.
I20240502 14:04:55.728129 22122 leader_election.cc:302] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters: 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:55.728936 22366 raft_consensus.cc:2781] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.729604 22366 raft_consensus.cc:686] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:55.730454 22366 consensus_queue.cc:237] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.731678 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:55.732326 22232 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 0.
I20240502 14:04:55.733561 22270 leader_election.cc:302] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:04:55.734352 22358 raft_consensus.cc:2781] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.734712 22358 raft_consensus.cc:482] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.735018 22358 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.741806 22358 raft_consensus.cc:504] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.744319 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:55.745098 22358 leader_election.cc:288] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.745277 22156 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 0.
I20240502 14:04:55.753068 22350 tablet_bootstrap.cc:492] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.753556 22350 ts_tablet_manager.cc:1367] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.133s user 0.051s sys 0.048s
I20240502 14:04:55.756196 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:55.756734 22156 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.756405 22350 raft_consensus.cc:348] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.757166 22350 raft_consensus.cc:374] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.757637 22350 raft_consensus.cc:729] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.757802 22197 leader_election.cc:302] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:55.758889 22352 raft_consensus.cc:2781] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.758608 22350 consensus_queue.cc:260] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.759271 22352 raft_consensus.cc:482] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.759985 22352 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.760401 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:55.751586 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:55.750463 22155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:55.764380 22306 raft_consensus.cc:2379] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c3a0e50105649c3b949b797345e463e in current term 1: Already voted for candidate 6151fcb16d6d4d01b2ea65767bda6076 in this term.
I20240502 14:04:55.766672 22358 raft_consensus.cc:482] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.767130 22358 raft_consensus.cc:504] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.768514 22156 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 1.
I20240502 14:04:55.769039 22358 leader_election.cc:288] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.770787 22272 leader_election.cc:302] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:55.771384 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f546cfa74964958816fa7b370229792" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
W20240502 14:04:55.773295 22270 leader_election.cc:341] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:55.773432 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f546cfa74964958816fa7b370229792" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:55.774235 22156 raft_consensus.cc:2454] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 0.
I20240502 14:04:55.775293 22272 leader_election.cc:302] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters: 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:55.772156 22358 raft_consensus.cc:2781] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.780011 22350 ts_tablet_manager.cc:1398] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.026s user 0.006s sys 0.002s
I20240502 14:04:55.778800 22352 raft_consensus.cc:504] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.782935 22350 tablet_bootstrap.cc:492] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.783531 22232 raft_consensus.cc:2379] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6151fcb16d6d4d01b2ea65767bda6076 in current term 1: Already voted for candidate 8c3a0e50105649c3b949b797345e463e in this term.
I20240502 14:04:55.784845 22352 leader_election.cc:288] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:55.797032 22387 raft_consensus.cc:2781] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.797420 22387 raft_consensus.cc:482] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.797760 22387 raft_consensus.cc:3037] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.799059 22350 tablet_bootstrap.cc:654] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.798897 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:55.802284 22358 raft_consensus.cc:686] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:55.802295 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:55.803383 22156 raft_consensus.cc:2379] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c3a0e50105649c3b949b797345e463e in current term 1: Already voted for candidate 6151fcb16d6d4d01b2ea65767bda6076 in this term.
I20240502 14:04:55.810683 22052 catalog_manager.cc:5484] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 0 to 1, leader changed from <none> to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 1 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:55.812366 22349 tablet_bootstrap.cc:654] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.803092 22358 consensus_queue.cc:237] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.804105 22387 raft_consensus.cc:504] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.805248 22353 tablet_bootstrap.cc:492] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.816948 22353 ts_tablet_manager.cc:1367] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.106s user 0.024s sys 0.018s
I20240502 14:04:55.817376 22387 leader_election.cc:288] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.818344 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f546cfa74964958816fa7b370229792" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
W20240502 14:04:55.819478 22270 leader_election.cc:341] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:55.819562 22353 raft_consensus.cc:348] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.820283 22353 raft_consensus.cc:374] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.820574 22353 raft_consensus.cc:729] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.821230 22353 consensus_queue.cc:260] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.824142 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f546cfa74964958816fa7b370229792" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:55.824762 22156 raft_consensus.cc:3037] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.827505 22197 leader_election.cc:302] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e; no voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c
I20240502 14:04:55.828225 22352 raft_consensus.cc:2726] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:04:55.829810 22156 raft_consensus.cc:2454] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 1.
I20240502 14:04:55.830999 22272 leader_election.cc:302] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters: 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:55.807250 22350 tablet_bootstrap.cc:492] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.831815 22358 raft_consensus.cc:2781] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.832226 22353 ts_tablet_manager.cc:1398] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240502 14:04:55.832382 22358 raft_consensus.cc:686] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:55.832700 22350 ts_tablet_manager.cc:1367] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.000s
I20240502 14:04:55.833479 22366 raft_consensus.cc:482] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.833319 22358 consensus_queue.cc:237] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.833909 22366 raft_consensus.cc:504] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.835266 22350 raft_consensus.cc:348] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.835754 22366 leader_election.cc:288] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.835971 22350 raft_consensus.cc:374] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.836259 22350 raft_consensus.cc:729] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.836865 22350 consensus_queue.cc:260] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.837237 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:55.837733 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:55.837877 22232 raft_consensus.cc:2454] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 0.
I20240502 14:04:55.839021 22120 leader_election.cc:302] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:55.839803 22353 tablet_bootstrap.cc:492] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
W20240502 14:04:55.841006 22122 leader_election.cc:341] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:55.843396 22366 raft_consensus.cc:2781] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.843770 22366 raft_consensus.cc:482] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.844175 22366 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.839011 22350 ts_tablet_manager.cc:1398] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:04:55.846974 22350 tablet_bootstrap.cc:492] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.852815 22350 tablet_bootstrap.cc:654] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.856644 22353 tablet_bootstrap.cc:654] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.865604 22353 tablet_bootstrap.cc:492] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.866039 22353 ts_tablet_manager.cc:1367] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.000s
I20240502 14:04:55.866750 22052 catalog_manager.cc:5484] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:55.868764 22353 raft_consensus.cc:348] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.870112 22353 raft_consensus.cc:374] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.870424 22353 raft_consensus.cc:729] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.871119 22353 consensus_queue.cc:260] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.869290 22366 raft_consensus.cc:504] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.873980 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:55.874538 22353 ts_tablet_manager.cc:1398] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
W20240502 14:04:55.875437 22122 leader_election.cc:341] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:55.877506 22353 tablet_bootstrap.cc:492] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.883776 22366 leader_election.cc:288] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Requested vote from peers 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.884744 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:55.885938 22232 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.891255 22350 tablet_bootstrap.cc:492] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.891358 22353 tablet_bootstrap.cc:654] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.891760 22350 ts_tablet_manager.cc:1367] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.005s
I20240502 14:04:55.891994 22232 raft_consensus.cc:2454] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 1.
I20240502 14:04:55.893112 22120 leader_election.cc:302] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters: 6151fcb16d6d4d01b2ea65767bda6076
I20240502 14:04:55.893837 22366 raft_consensus.cc:2781] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.894315 22366 raft_consensus.cc:686] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:55.894412 22350 raft_consensus.cc:348] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.895053 22350 raft_consensus.cc:374] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.895362 22350 raft_consensus.cc:729] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.895166 22366 consensus_queue.cc:237] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.896023 22349 tablet_bootstrap.cc:492] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.895958 22350 consensus_queue.cc:260] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.896525 22349 ts_tablet_manager.cc:1367] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.172s user 0.041s sys 0.019s
I20240502 14:04:55.899051 22349 raft_consensus.cc:348] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.899767 22349 raft_consensus.cc:374] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.900046 22349 raft_consensus.cc:729] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.900506 22358 raft_consensus.cc:482] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.900954 22358 raft_consensus.cc:504] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.900745 22349 consensus_queue.cc:260] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.903219 22358 leader_election.cc:288] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.903533 22349 ts_tablet_manager.cc:1398] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20240502 14:04:55.903947 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:55.904369 22349 tablet_bootstrap.cc:492] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.904570 22156 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 0.
I20240502 14:04:55.905740 22272 leader_election.cc:302] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:55.905963 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:55.906498 22358 raft_consensus.cc:2781] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.906620 22232 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 0.
I20240502 14:04:55.906867 22358 raft_consensus.cc:482] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.907265 22358 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.907315 22052 catalog_manager.cc:5484] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:55.910214 22349 tablet_bootstrap.cc:654] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.913321 22358 raft_consensus.cc:504] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.916580 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:55.916996 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:55.917557 22156 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.917132 22232 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.919243 22358 leader_election.cc:288] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:55.928728 22232 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 1.
I20240502 14:04:55.929769 22270 leader_election.cc:302] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:04:55.930514 22358 raft_consensus.cc:2781] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.930969 22358 raft_consensus.cc:686] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:55.931586 22156 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 1.
I20240502 14:04:55.931744 22358 consensus_queue.cc:237] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.940090 22350 ts_tablet_manager.cc:1398] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.048s user 0.006s sys 0.006s
I20240502 14:04:55.940914 22350 tablet_bootstrap.cc:492] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c: Bootstrap starting.
I20240502 14:04:55.940734 22052 catalog_manager.cc:5484] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 0 to 1, leader changed from <none> to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 1 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:55.948073 22349 tablet_bootstrap.cc:492] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:55.948504 22349 ts_tablet_manager.cc:1367] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.018s
I20240502 14:04:55.948668 22353 tablet_bootstrap.cc:492] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:55.949198 22353 ts_tablet_manager.cc:1367] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.006s
I20240502 14:04:55.952098 22353 raft_consensus.cc:348] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.956372 22353 raft_consensus.cc:374] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.956703 22353 raft_consensus.cc:729] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.956934 22350 tablet_bootstrap.cc:654] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.953677 22349 raft_consensus.cc:348] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.957357 22353 consensus_queue.cc:260] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:55.958236 22349 raft_consensus.cc:374] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.959913 22353 ts_tablet_manager.cc:1398] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240502 14:04:55.958838 22349 raft_consensus.cc:729] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.960783 22353 tablet_bootstrap.cc:492] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076: Bootstrap starting.
I20240502 14:04:55.961498 22349 consensus_queue.cc:260] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.966251 22349 ts_tablet_manager.cc:1398] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.017s user 0.004s sys 0.004s
I20240502 14:04:55.967278 22349 tablet_bootstrap.cc:492] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:55.970252 22353 tablet_bootstrap.cc:654] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.976640 22350 tablet_bootstrap.cc:492] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c: No bootstrap required, opened a new log
I20240502 14:04:55.976667 22349 tablet_bootstrap.cc:654] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.977162 22350 ts_tablet_manager.cc:1367] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.008s
I20240502 14:04:55.966910 22050 catalog_manager.cc:5484] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:55.979578 22350 raft_consensus.cc:348] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.980239 22350 raft_consensus.cc:374] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.980530 22350 raft_consensus.cc:729] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.981149 22350 consensus_queue.cc:260] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:55.983549 22350 ts_tablet_manager.cc:1398] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:04:56.000556 22353 tablet_bootstrap.cc:492] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076: No bootstrap required, opened a new log
I20240502 14:04:56.000864 22353 ts_tablet_manager.cc:1367] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.008s
I20240502 14:04:56.002432 22396 raft_consensus.cc:482] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.003104 22358 raft_consensus.cc:482] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.002887 22353 raft_consensus.cc:348] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.003614 22353 raft_consensus.cc:374] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:56.003826 22349 tablet_bootstrap.cc:492] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:56.004014 22353 raft_consensus.cc:729] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Initialized, Role: FOLLOWER
I20240502 14:04:56.004483 22349 ts_tablet_manager.cc:1367] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.016s
I20240502 14:04:56.002923 22396 raft_consensus.cc:504] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.004812 22353 consensus_queue.cc:260] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.006237 22396 leader_election.cc:288] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.003573 22358 raft_consensus.cc:504] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.006927 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848dc17f220448abb94b378fb2999425" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:56.007297 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848dc17f220448abb94b378fb2999425" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:56.007645 22232 raft_consensus.cc:2454] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 0.
I20240502 14:04:56.007926 22306 raft_consensus.cc:2454] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 0.
I20240502 14:04:56.008293 22358 leader_election.cc:288] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:56.008867 22120 leader_election.cc:302] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.009677 22353 ts_tablet_manager.cc:1398] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:04:56.009577 22396 raft_consensus.cc:2781] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:56.010107 22396 raft_consensus.cc:482] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:56.010208 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:56.010495 22396 raft_consensus.cc:3037] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:04:56.011694 22270 leader_election.cc:341] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:56.009507 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:56.012730 22156 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 0.
I20240502 14:04:56.012600 22349 raft_consensus.cc:348] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.013446 22349 raft_consensus.cc:374] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:56.013772 22349 raft_consensus.cc:729] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:56.013873 22272 leader_election.cc:302] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters: 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:56.014626 22358 raft_consensus.cc:2781] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:56.015058 22358 raft_consensus.cc:482] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:56.014689 22349 consensus_queue.cc:260] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.015406 22358 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.016780 22349 ts_tablet_manager.cc:1398] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 14:04:56.017623 22349 tablet_bootstrap.cc:492] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e: Bootstrap starting.
I20240502 14:04:56.021265 22358 raft_consensus.cc:504] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.022307 22349 tablet_bootstrap.cc:654] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:56.024755 22358 leader_election.cc:288] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:56.025835 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:56.024015 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:56.026412 22232 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.026752 22156 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.027410 22396 raft_consensus.cc:504] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.033704 22349 tablet_bootstrap.cc:492] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e: No bootstrap required, opened a new log
I20240502 14:04:56.034113 22349 ts_tablet_manager.cc:1367] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240502 14:04:56.034860 22396 leader_election.cc:288] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.035820 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848dc17f220448abb94b378fb2999425" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:56.036705 22231 raft_consensus.cc:3037] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.036969 22349 raft_consensus.cc:348] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.037725 22349 raft_consensus.cc:374] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:56.037618 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848dc17f220448abb94b378fb2999425" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:56.038057 22349 raft_consensus.cc:729] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Initialized, Role: FOLLOWER
I20240502 14:04:56.038205 22306 raft_consensus.cc:3037] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.038666 22349 consensus_queue.cc:260] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.040815 22349 ts_tablet_manager.cc:1398] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:04:56.043300 22231 raft_consensus.cc:2454] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 1.
I20240502 14:04:56.044461 22120 leader_election.cc:302] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.045034 22396 raft_consensus.cc:2781] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:56.045521 22396 raft_consensus.cc:686] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:56.046427 22306 raft_consensus.cc:2454] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 1.
I20240502 14:04:56.046295 22396 consensus_queue.cc:237] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.032445 22156 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 1.
I20240502 14:04:56.032660 22232 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 1.
I20240502 14:04:56.048444 22272 leader_election.cc:302] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.050108 22358 raft_consensus.cc:2781] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:56.050544 22358 raft_consensus.cc:686] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:56.051286 22358 consensus_queue.cc:237] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.055658 22050 catalog_manager.cc:5484] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 0 to 1, leader changed from <none> to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 1 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:56.059247 22051 catalog_manager.cc:5484] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:56.100239 22352 raft_consensus.cc:482] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.100798 22352 raft_consensus.cc:504] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.102865 22352 leader_election.cc:288] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.103660 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:56.103909 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:56.104164 22156 raft_consensus.cc:2454] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 0.
I20240502 14:04:56.104568 22306 raft_consensus.cc:2454] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 0.
I20240502 14:04:56.104983 22197 leader_election.cc:302] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.105659 22352 raft_consensus.cc:2781] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:56.106022 22352 raft_consensus.cc:482] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:56.106335 22352 raft_consensus.cc:3037] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.112416 22352 raft_consensus.cc:504] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.114148 22352 leader_election.cc:288] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.115173 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:56.115212 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:56.115916 22156 raft_consensus.cc:3037] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.116007 22306 raft_consensus.cc:3037] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.119688 22156 raft_consensus.cc:2454] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 1.
I20240502 14:04:56.120465 22197 leader_election.cc:302] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.121011 22352 raft_consensus.cc:2781] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:56.121837 22306 raft_consensus.cc:2454] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 1.
I20240502 14:04:56.121912 22352 raft_consensus.cc:686] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:04:56.122866 22352 consensus_queue.cc:237] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.129761 22051 catalog_manager.cc:5484] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 1 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:56.169095 22358 raft_consensus.cc:482] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.169458 22358 raft_consensus.cc:504] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.171013 22358 leader_election.cc:288] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:56.172137 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:56.172408 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:56.173183 22232 raft_consensus.cc:2379] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6151fcb16d6d4d01b2ea65767bda6076 in current term 1: Already voted for candidate eef3139068244521a464f0100bc1d29c in this term.
I20240502 14:04:56.173976 22270 leader_election.cc:302] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076; no voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c
I20240502 14:04:56.174548 22358 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.178748 22366 consensus_queue.cc:1035] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.180603 22358 raft_consensus.cc:2726] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:56.185591 22366 consensus_queue.cc:1035] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.205390 22358 consensus_queue.cc:1035] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.220278 22352 raft_consensus.cc:482] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.220758 22352 raft_consensus.cc:504] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.222618 22352 leader_election.cc:288] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.224249 22155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f546cfa74964958816fa7b370229792" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:56.225029 22155 raft_consensus.cc:2379] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c3a0e50105649c3b949b797345e463e in current term 1: Already voted for candidate 6151fcb16d6d4d01b2ea65767bda6076 in this term.
I20240502 14:04:56.226418 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f546cfa74964958816fa7b370229792" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076" is_pre_election: true
I20240502 14:04:56.228081 22197 leader_election.cc:302] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e; no voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c
I20240502 14:04:56.232625 22352 raft_consensus.cc:3037] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.246801 22387 consensus_queue.cc:1035] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.241168 22352 raft_consensus.cc:2726] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:56.258849 22305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
mode: GRACEFUL
new_leader_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:50688
I20240502 14:04:56.260092 22305 raft_consensus.cc:595] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Received request to transfer leadership to TS eef3139068244521a464f0100bc1d29c
I20240502 14:04:56.290282 22156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "669279922bc24f55908b209483dd8271"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:44550
I20240502 14:04:56.290746 22156 raft_consensus.cc:595] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:56.295300 22231 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.298743 22305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:50688
I20240502 14:04:56.299351 22305 raft_consensus.cc:595] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:56.301173 22357 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.303714 22305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:50688
I20240502 14:04:56.304266 22305 raft_consensus.cc:595] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:04:56.312186 22357 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.319088 22357 raft_consensus.cc:979] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:04:56.319497 22357 raft_consensus.cc:1067] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:04:56.323652 22232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "669279922bc24f55908b209483dd8271"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49158
I20240502 14:04:56.324549 22232 raft_consensus.cc:482] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:04:56.324924 22232 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.330675 22232 raft_consensus.cc:504] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.338268 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:56.338855 22156 raft_consensus.cc:3032] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:04:56.339174 22156 raft_consensus.cc:729] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:56.339931 22156 consensus_queue.cc:260] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.340580 22417 raft_consensus.cc:482] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.341053 22417 raft_consensus.cc:504] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:56.342562 22387 consensus_queue.cc:1035] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.343318 22417 leader_election.cc:288] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:56.343924 22155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef3139068244521a464f0100bc1d29c" is_pre_election: true
I20240502 14:04:56.345425 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3a0e50105649c3b949b797345e463e" is_pre_election: true
I20240502 14:04:56.346024 22231 raft_consensus.cc:2379] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6151fcb16d6d4d01b2ea65767bda6076 in current term 1: Already voted for candidate eef3139068244521a464f0100bc1d29c in this term.
I20240502 14:04:56.347018 22270 leader_election.cc:302] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076; no voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c
I20240502 14:04:56.348135 22417 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.340965 22156 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.356639 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:56.357256 22305 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.357463 22156 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:04:56.367023 22305 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:04:56.367508 22232 leader_election.cc:288] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.370569 22197 leader_election.cc:302] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:04:56.371328 22352 raft_consensus.cc:2781] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:04:56.371778 22352 raft_consensus.cc:686] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:04:56.372560 22352 consensus_queue.cc:237] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.376477 22417 raft_consensus.cc:2726] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:56.377216 22417 consensus_queue.cc:1035] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.377830 22366 consensus_queue.cc:1035] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.386528 22052 catalog_manager.cc:5484] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 1 to 2, leader changed from eef3139068244521a464f0100bc1d29c (127.21.129.1) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:56.395464 22396 consensus_queue.cc:1035] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.404208 22417 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.417511 22387 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.428517 22358 raft_consensus.cc:979] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: : Instructing follower eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:04:56.429119 22358 raft_consensus.cc:1067] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Signalling peer eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:04:56.430635 22156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:44542
I20240502 14:04:56.432991 22156 raft_consensus.cc:482] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:04:56.433372 22156 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.438542 22156 raft_consensus.cc:504] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.440251 22156 leader_election.cc:288] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 2 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.440621 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:56.440842 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:56.441182 22232 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.441402 22305 raft_consensus.cc:3032] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:04:56.441735 22305 raft_consensus.cc:729] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:56.442304 22305 consensus_queue.cc:260] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.443131 22305 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.446959 22305 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 2.
I20240502 14:04:56.447257 22232 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 2.
I20240502 14:04:56.447804 22122 leader_election.cc:302] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.448482 22396 raft_consensus.cc:2781] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:04:56.448941 22396 raft_consensus.cc:686] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:56.449712 22396 consensus_queue.cc:237] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.456070 22050 catalog_manager.cc:5484] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 1 to 2, leader changed from 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3) to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 2 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:56.587282 22387 consensus_queue.cc:1035] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.598215 22396 consensus_queue.cc:1035] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.598608 22388 consensus_queue.cc:1035] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.605862 22396 consensus_queue.cc:1035] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.622249 22387 raft_consensus.cc:979] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:04:56.622704 22388 raft_consensus.cc:1067] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:04:56.624336 22231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49172
I20240502 14:04:56.624926 22231 raft_consensus.cc:482] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:04:56.625259 22231 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.632190 22231 raft_consensus.cc:504] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.639529 22231 leader_election.cc:288] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.641781 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:56.642354 22156 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.647522 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:56.648098 22305 raft_consensus.cc:3032] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:04:56.648383 22305 raft_consensus.cc:729] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:56.648981 22305 consensus_queue.cc:260] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.649849 22305 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.652436 22156 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:04:56.653388 22197 leader_election.cc:302] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.654139 22352 raft_consensus.cc:2781] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:04:56.654613 22352 raft_consensus.cc:686] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:04:56.654875 22305 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:04:56.655440 22352 consensus_queue.cc:237] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.661449 22050 catalog_manager.cc:5484] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 1 to 2, leader changed from 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:56.676146 22352 consensus_queue.cc:1035] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.691572 22352 consensus_queue.cc:1035] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.799921 22388 raft_consensus.cc:979] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:04:56.800405 22358 raft_consensus.cc:1067] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:04:56.802026 22231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49172
I20240502 14:04:56.802598 22231 raft_consensus.cc:482] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:04:56.802944 22231 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.807353 22231 raft_consensus.cc:504] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.808687 22231 leader_election.cc:288] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:04:56.809486 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:56.809854 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:04:56.810067 22156 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.810411 22305 raft_consensus.cc:3032] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:04:56.810742 22305 raft_consensus.cc:729] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:56.811494 22305 consensus_queue.cc:260] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.812700 22305 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:56.816196 22156 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:04:56.817284 22197 leader_election.cc:302] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:56.818109 22433 raft_consensus.cc:2781] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:04:56.818616 22433 raft_consensus.cc:686] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:04:56.818743 22305 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:04:56.819543 22433 consensus_queue.cc:237] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:56.826392 22051 catalog_manager.cc:5484] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 1 to 2, leader changed from 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:56.841442 22232 raft_consensus.cc:1261] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:56.842864 22396 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:56.852628 22305 raft_consensus.cc:1261] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:56.855476 22435 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:56.903237 22305 raft_consensus.cc:1261] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:56.904543 22433 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:56.914675 22156 raft_consensus.cc:1261] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:56.916795 22407 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:57.093489 22305 raft_consensus.cc:1261] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:57.094610 22407 consensus_queue.cc:1035] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:57.101691 22156 raft_consensus.cc:1261] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:57.103623 22436 consensus_queue.cc:1035] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:57.280763 22305 raft_consensus.cc:1261] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:57.281752 22407 consensus_queue.cc:1035] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:57.292311 22156 raft_consensus.cc:1261] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:04:57.293936 22436 consensus_queue.cc:1035] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:59.328130 22020 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240502 14:04:59.328814 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:59.350601 22231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "669279922bc24f55908b209483dd8271"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
mode: GRACEFUL
new_leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:49188
I20240502 14:04:59.351240 22231 raft_consensus.cc:595] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Received request to transfer leadership to TS 6151fcb16d6d4d01b2ea65767bda6076
I20240502 14:04:59.355108 22433 raft_consensus.cc:979] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: : Instructing follower 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:04:59.355594 22445 raft_consensus.cc:1067] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Signalling peer 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:04:59.357054 22306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "669279922bc24f55908b209483dd8271"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:50686
I20240502 14:04:59.357476 22306 raft_consensus.cc:482] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:04:59.357782 22306 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:04:59.363337 22306 raft_consensus.cc:504] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:59.363559 22156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
mode: GRACEFUL
new_leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:44562
I20240502 14:04:59.364022 22156 raft_consensus.cc:595] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Received request to transfer leadership to TS 6151fcb16d6d4d01b2ea65767bda6076
I20240502 14:04:59.364856 22306 leader_election.cc:288] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 3 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:59.364984 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:04:59.365509 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:59.365468 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:59.366123 22156 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:04:59.366186 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:59.366678 22231 raft_consensus.cc:3032] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:04:59.366915 22231 raft_consensus.cc:729] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:04:59.367504 22231 consensus_queue.cc:260] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:59.368346 22231 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:04:59.370065 22156 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 3.
I20240502 14:04:59.370884 22272 leader_election.cc:302] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:04:59.371518 22358 raft_consensus.cc:2781] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:04:59.372633 22358 raft_consensus.cc:686] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:59.372826 22231 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 3.
I20240502 14:04:59.373440 22358 consensus_queue.cc:237] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:04:59.380043 22051 catalog_manager.cc:5484] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 2 to 3, leader changed from 8c3a0e50105649c3b949b797345e463e (127.21.129.2) to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 3 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:59.835050 22440 raft_consensus.cc:979] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: : Instructing follower 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:04:59.835495 22453 raft_consensus.cc:1067] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Signalling peer 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:04:59.836808 22306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:50682
I20240502 14:04:59.837258 22306 raft_consensus.cc:482] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:04:59.837531 22306 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:59.841610 22306 raft_consensus.cc:504] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:59.842990 22306 leader_election.cc:288] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:04:59.843632 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:04:59.843936 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:04:59.844256 22156 raft_consensus.cc:3032] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:04:59.844501 22231 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:59.844597 22156 raft_consensus.cc:729] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:04:59.845706 22156 consensus_queue.cc:260] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:59.846503 22156 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:59.848789 22231 raft_consensus.cc:2454] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 2.
I20240502 14:04:59.849539 22232 raft_consensus.cc:1226] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Rejecting Update request from peer eef3139068244521a464f0100bc1d29c for earlier term 1. Current term is 2. Ops: []
I20240502 14:04:59.850318 22270 leader_election.cc:302] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:04:59.851377 22358 raft_consensus.cc:2781] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:04:59.851840 22358 raft_consensus.cc:686] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:04:59.851838 22156 raft_consensus.cc:2454] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 2.
I20240502 14:04:59.852613 22358 consensus_queue.cc:237] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:04:59.859077 22051 catalog_manager.cc:5484] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 1 to 2, leader changed from eef3139068244521a464f0100bc1d29c (127.21.129.1) to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 2 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:59.870116 22156 raft_consensus.cc:1261] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:04:59.871014 22358 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:04:59.878741 22232 raft_consensus.cc:1261] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:04:59.879971 22358 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:00.344062 22156 raft_consensus.cc:1261] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:00.344944 22358 consensus_queue.cc:1035] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:00.355369 22231 raft_consensus.cc:1261] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:00.357089 22461 consensus_queue.cc:1035] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:00.366513 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:00.372012 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:00.372361 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:01.373193 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:01.378376 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:01.378759 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:02.379472 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:02.385187 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:02.385530 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:03.386345 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:03.392812 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:03.393146 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:04.393806 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:04.398221 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:04.398597 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:05.399294 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:05.403934 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:05.404289 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:06.404984 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:06.410007 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:06.410439 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:07.411180 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:07.415760 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:07.416116 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:08.416806 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:08.421415 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:08.421758 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:09.441486 22020 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240502 14:05:09.445351 22306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:50688
I20240502 14:05:09.445984 22306 raft_consensus.cc:595] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:05:09.450321 22154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:44550
I20240502 14:05:09.450763 22154 raft_consensus.cc:595] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:05:09.454550 22154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "848dc17f220448abb94b378fb2999425"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:44550
I20240502 14:05:09.454970 22154 raft_consensus.cc:595] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:05:09.458719 22154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:44550
I20240502 14:05:09.459129 22154 raft_consensus.cc:595] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:05:09.462930 22306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "669279922bc24f55908b209483dd8271"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
mode: GRACEFUL
new_leader_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:50688
I20240502 14:05:09.463356 22306 raft_consensus.cc:595] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Received request to transfer leadership to TS 8c3a0e50105649c3b949b797345e463e
I20240502 14:05:09.499038 22472 raft_consensus.cc:979] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.499728 22453 raft_consensus.cc:1067] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.501379 22232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49158
I20240502 14:05:09.501842 22232 raft_consensus.cc:482] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:09.502137 22232 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:09.507104 22232 raft_consensus.cc:504] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.508972 22232 leader_election.cc:288] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 3 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:09.509910 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:09.509864 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:09.510644 22156 raft_consensus.cc:3032] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:09.510811 22306 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:09.510955 22156 raft_consensus.cc:729] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:05:09.511737 22156 consensus_queue.cc:260] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.512672 22156 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:09.515085 22306 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 3.
I20240502 14:05:09.516079 22197 leader_election.cc:302] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:05:09.516741 22469 raft_consensus.cc:2781] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:09.516989 22156 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 3.
I20240502 14:05:09.517721 22469 raft_consensus.cc:686] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:09.518601 22469 consensus_queue.cc:237] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.525278 22050 catalog_manager.cc:5484] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 2 to 3, leader changed from eef3139068244521a464f0100bc1d29c (127.21.129.1) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 3 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:09.895750 22479 raft_consensus.cc:979] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.896085 22479 raft_consensus.cc:1067] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.898110 22231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "669279922bc24f55908b209483dd8271"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49172
I20240502 14:05:09.898655 22231 raft_consensus.cc:482] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:09.899008 22231 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:09.905845 22231 raft_consensus.cc:504] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.907748 22231 leader_election.cc:288] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 4 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:09.911526 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:09.911859 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669279922bc24f55908b209483dd8271" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:09.912154 22156 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:09.912520 22305 raft_consensus.cc:3032] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:05:09.912847 22305 raft_consensus.cc:729] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:05:09.913653 22305 consensus_queue.cc:260] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.914788 22305 raft_consensus.cc:3037] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:09.917919 22156 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 4.
I20240502 14:05:09.921608 22305 raft_consensus.cc:2454] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 4.
I20240502 14:05:09.923099 22197 leader_election.cc:302] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:05:09.924062 22469 raft_consensus.cc:2781] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:05:09.924497 22469 raft_consensus.cc:686] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 4 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:09.924986 22479 raft_consensus.cc:979] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.925158 22472 raft_consensus.cc:979] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.925311 22479 raft_consensus.cc:1067] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.925335 22469 consensus_queue.cc:237] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.928560 22472 raft_consensus.cc:1067] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.930809 22472 raft_consensus.cc:979] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: : Instructing follower 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.931317 22472 raft_consensus.cc:1067] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Signalling peer 8c3a0e50105649c3b949b797345e463e to start an election
I20240502 14:05:09.932401 22232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "848dc17f220448abb94b378fb2999425"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49158
I20240502 14:05:09.932766 22230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49172
I20240502 14:05:09.932834 22232 raft_consensus.cc:482] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:09.933357 22232 raft_consensus.cc:3037] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:09.933598 22229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
from {username='slave'} at 127.0.0.1:49158
I20240502 14:05:09.934844 22229 raft_consensus.cc:482] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:09.935281 22229 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:09.937247 22050 catalog_manager.cc:5484] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 3 to 4, leader changed from 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 4 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:09.933362 22230 raft_consensus.cc:482] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:09.939659 22230 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:09.940718 22229 raft_consensus.cc:504] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.942543 22229 leader_election.cc:288] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:09.944610 22304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:09.944795 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:09.945432 22304 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:09.945523 22156 raft_consensus.cc:3032] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:09.946389 22156 raft_consensus.cc:729] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:05:09.946906 22156 consensus_queue.cc:260] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.947865 22156 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:09.948684 22232 raft_consensus.cc:504] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.950259 22232 leader_election.cc:288] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:09.951397 22156 raft_consensus.cc:2454] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:05:09.951678 22304 raft_consensus.cc:2454] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:05:09.953883 22230 raft_consensus.cc:504] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:05:09.955226 22197 leader_election.cc:302] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:05:09.955327 22230 leader_election.cc:288] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 3 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:09.955530 22304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848dc17f220448abb94b378fb2999425" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:09.955685 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848dc17f220448abb94b378fb2999425" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:09.956072 22304 raft_consensus.cc:3037] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:09.956267 22156 raft_consensus.cc:3032] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:09.956297 22473 raft_consensus.cc:2781] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:09.956775 22156 raft_consensus.cc:729] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:05:09.957067 22473 raft_consensus.cc:686] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:09.957499 22154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:09.957572 22156 consensus_queue.cc:260] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.958145 22154 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:09.958086 22473 consensus_queue.cc:237] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.958906 22156 raft_consensus.cc:3037] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:09.962379 22304 raft_consensus.cc:2454] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:05:09.964395 22156 raft_consensus.cc:2454] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 2.
I20240502 14:05:09.964512 22154 raft_consensus.cc:2454] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 3.
I20240502 14:05:09.964536 22197 leader_election.cc:302] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, 8c3a0e50105649c3b949b797345e463e; no voters:
I20240502 14:05:09.965481 22473 raft_consensus.cc:2781] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:09.965899 22473 raft_consensus.cc:686] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:09.965992 22197 leader_election.cc:302] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:05:09.966727 22469 raft_consensus.cc:2781] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:09.966710 22473 consensus_queue.cc:237] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:09.968102 22304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d30859f04ad89ac47caa0cb53622" candidate_uuid: "8c3a0e50105649c3b949b797345e463e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:09.968660 22304 raft_consensus.cc:3032] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:09.968968 22304 raft_consensus.cc:729] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:05:09.969844 22304 consensus_queue.cc:260] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:05:09.971051 22304 raft_consensus.cc:3037] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:09.971859 22469 raft_consensus.cc:686] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Becoming Leader. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:09.971254 22050 catalog_manager.cc:5484] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 1 to 2, leader changed from eef3139068244521a464f0100bc1d29c (127.21.129.1) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:09.972762 22469 consensus_queue.cc:237] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } }
I20240502 14:05:09.977398 22304 raft_consensus.cc:2454] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3a0e50105649c3b949b797345e463e in term 3.
I20240502 14:05:09.984673 22050 catalog_manager.cc:5484] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 1 to 2, leader changed from eef3139068244521a464f0100bc1d29c (127.21.129.1) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:09.985656 22050 catalog_manager.cc:5484] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e reported cstate change: term changed from 2 to 3, leader changed from 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3) to 8c3a0e50105649c3b949b797345e463e (127.21.129.2). New cstate: current_term: 3 leader_uuid: "8c3a0e50105649c3b949b797345e463e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:09.997126 22156 raft_consensus.cc:1261] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:09.998158 22473 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:10.006959 22304 raft_consensus.cc:1261] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:10.009307 22469 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:10.326893 22156 raft_consensus.cc:1261] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:10.327932 22469 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:10.336671 22304 raft_consensus.cc:1261] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:10.338104 22473 consensus_queue.cc:1035] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:10.378935 22156 raft_consensus.cc:1261] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:10.379866 22473 consensus_queue.cc:1035] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:10.388857 22304 raft_consensus.cc:1261] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:10.390025 22469 consensus_queue.cc:1035] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:10.398754 22304 raft_consensus.cc:1261] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:10.399931 22469 consensus_queue.cc:1035] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:10.410173 22156 raft_consensus.cc:1261] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:10.411262 22469 consensus_queue.cc:1035] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:10.514580 22156 raft_consensus.cc:1261] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:10.515501 22473 consensus_queue.cc:1035] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:10.522497 22305 raft_consensus.cc:1261] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Refusing update from remote peer 8c3a0e50105649c3b949b797345e463e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:10.524287 22493 consensus_queue.cc:1035] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:05:10.867614 22327 debug-util.cc:398] Leaking SignalData structure 0x7b0800177060 after lost signal to thread 22023
W20240502 14:05:10.868383 22327 debug-util.cc:398] Leaking SignalData structure 0x7b0800177740 after lost signal to thread 22087
W20240502 14:05:10.869093 22327 debug-util.cc:398] Leaking SignalData structure 0x7b0800155c60 after lost signal to thread 22180
W20240502 14:05:10.869742 22327 debug-util.cc:398] Leaking SignalData structure 0x7b0800084cc0 after lost signal to thread 22256
W20240502 14:05:10.870385 22327 debug-util.cc:398] Leaking SignalData structure 0x7b08001ebba0 after lost signal to thread 22330
I20240502 14:05:12.760778 22020 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240502 14:05:12.761430 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:12.767634 22230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
mode: GRACEFUL
new_leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:49188
I20240502 14:05:12.768091 22230 raft_consensus.cc:595] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Received request to transfer leadership to TS 6151fcb16d6d4d01b2ea65767bda6076
I20240502 14:05:12.770095 22230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
mode: GRACEFUL
new_leader_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:49188
I20240502 14:05:12.770509 22230 raft_consensus.cc:595] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Received request to transfer leadership to TS eef3139068244521a464f0100bc1d29c
I20240502 14:05:12.772433 22230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
mode: GRACEFUL
new_leader_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:49188
I20240502 14:05:12.772940 22230 raft_consensus.cc:595] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Received request to transfer leadership to TS eef3139068244521a464f0100bc1d29c
I20240502 14:05:12.774488 22500 raft_consensus.cc:979] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: : Instructing follower eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:05:12.774888 22499 raft_consensus.cc:1067] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Signalling peer eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:05:12.775363 22230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
mode: GRACEFUL
new_leader_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:49188
I20240502 14:05:12.775935 22230 raft_consensus.cc:595] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Received request to transfer leadership to TS eef3139068244521a464f0100bc1d29c
I20240502 14:05:12.776160 22156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:44526
I20240502 14:05:12.776647 22156 raft_consensus.cc:482] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:12.776965 22156 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:12.777803 22230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f"
dest_uuid: "8c3a0e50105649c3b949b797345e463e"
mode: GRACEFUL
new_leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:49188
I20240502 14:05:12.778215 22230 raft_consensus.cc:595] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 LEADER]: Received request to transfer leadership to TS 6151fcb16d6d4d01b2ea65767bda6076
I20240502 14:05:12.778987 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 14:05:12.779408 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:12.782688 22156 raft_consensus.cc:504] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.785252 22156 leader_election.cc:288] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 3 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:12.785523 22230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:05:12.785858 22304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491e9d03d7da4dd3a9ab09767574a167" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:12.786348 22230 raft_consensus.cc:3032] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:12.786415 22304 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:12.786787 22230 raft_consensus.cc:729] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:12.787454 22230 consensus_queue.cc:260] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.788307 22230 raft_consensus.cc:3037] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:12.791679 22500 raft_consensus.cc:979] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: : Instructing follower eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:05:12.792094 22499 raft_consensus.cc:1067] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Signalling peer eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:05:12.792531 22304 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 3.
I20240502 14:05:12.793692 22156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:44526
I20240502 14:05:12.793725 22122 leader_election.cc:302] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:05:12.794308 22156 raft_consensus.cc:482] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:12.794649 22156 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:12.794742 22230 raft_consensus.cc:2454] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 3.
I20240502 14:05:12.796298 22501 raft_consensus.cc:2781] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:12.797333 22501 raft_consensus.cc:686] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 3 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:05:12.798671 22501 consensus_queue.cc:237] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.800593 22156 raft_consensus.cc:504] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.802791 22230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:05:12.803227 22156 leader_election.cc:288] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 4 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:12.803326 22230 raft_consensus.cc:3032] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:05:12.803647 22230 raft_consensus.cc:729] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:12.804201 22304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7d83ca7ba84da093d1d973c934eb95" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:12.804348 22230 consensus_queue.cc:260] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.804749 22304 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:12.805249 22230 raft_consensus.cc:3037] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:12.808804 22051 catalog_manager.cc:5484] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 2 to 3, leader changed from 8c3a0e50105649c3b949b797345e463e (127.21.129.2) to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 3 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:12.812582 22230 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 4.
I20240502 14:05:12.813474 22304 raft_consensus.cc:2454] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 4.
I20240502 14:05:12.813783 22120 leader_election.cc:302] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c3a0e50105649c3b949b797345e463e, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:05:12.814518 22501 raft_consensus.cc:2781] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:05:12.815181 22501 raft_consensus.cc:686] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 4 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:05:12.815842 22501 consensus_queue.cc:237] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.827140 22051 catalog_manager.cc:5484] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 3 to 4, leader changed from 8c3a0e50105649c3b949b797345e463e (127.21.129.2) to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 4 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:12.949898 22499 raft_consensus.cc:979] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: : Instructing follower 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:05:12.950278 22500 raft_consensus.cc:1067] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 LEADER]: Signalling peer 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:05:12.951444 22304 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:50686
I20240502 14:05:12.951854 22304 raft_consensus.cc:482] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:12.952133 22304 raft_consensus.cc:3037] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:12.956586 22304 raft_consensus.cc:504] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.958806 22304 leader_election.cc:288] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 2 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:05:12.959045 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:12.959374 22230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d091765cb37b415ba7ebbde7e534576f" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:05:12.959614 22156 raft_consensus.cc:3037] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:12.959920 22230 raft_consensus.cc:3032] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:12.960229 22230 raft_consensus.cc:729] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:12.960976 22230 consensus_queue.cc:260] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.962167 22230 raft_consensus.cc:3037] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:12.965093 22156 raft_consensus.cc:2454] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 2.
I20240502 14:05:12.966095 22272 leader_election.cc:302] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:05:12.966784 22479 raft_consensus.cc:2781] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:12.967572 22230 raft_consensus.cc:2454] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 2.
I20240502 14:05:12.968199 22479 raft_consensus.cc:686] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:05:12.969045 22479 consensus_queue.cc:237] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:12.975113 22051 catalog_manager.cc:5484] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 1 to 2, leader changed from 8c3a0e50105649c3b949b797345e463e (127.21.129.2) to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 2 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:13.109241 22499 raft_consensus.cc:979] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: : Instructing follower 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:05:13.109910 22500 raft_consensus.cc:1067] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Signalling peer 6151fcb16d6d4d01b2ea65767bda6076 to start an election
I20240502 14:05:13.111328 22304 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409"
dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
from {username='slave'} at 127.0.0.1:50686
I20240502 14:05:13.111877 22304 raft_consensus.cc:482] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:13.112176 22304 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:13.117925 22304 raft_consensus.cc:504] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:13.119602 22304 leader_election.cc:288] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 3 election: Requested vote from peers eef3139068244521a464f0100bc1d29c (127.21.129.1:43373), 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501)
I20240502 14:05:13.121206 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eef3139068244521a464f0100bc1d29c"
I20240502 14:05:13.121636 22230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172d1f7d66f54633a1423f26d55ba409" candidate_uuid: "6151fcb16d6d4d01b2ea65767bda6076" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:05:13.121932 22156 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:13.122198 22230 raft_consensus.cc:3032] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:13.122519 22230 raft_consensus.cc:729] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:13.123327 22230 consensus_queue.cc:260] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:13.124429 22230 raft_consensus.cc:3037] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:13.126375 22156 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 3.
I20240502 14:05:13.127353 22272 leader_election.cc:302] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:05:13.128238 22479 raft_consensus.cc:2781] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:13.128703 22479 raft_consensus.cc:686] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Becoming Leader. State: Replica: 6151fcb16d6d4d01b2ea65767bda6076, State: Running, Role: LEADER
I20240502 14:05:13.129639 22230 raft_consensus.cc:2454] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6151fcb16d6d4d01b2ea65767bda6076 in term 3.
I20240502 14:05:13.129854 22479 consensus_queue.cc:237] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:13.137985 22051 catalog_manager.cc:5484] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 reported cstate change: term changed from 2 to 3, leader changed from 8c3a0e50105649c3b949b797345e463e (127.21.129.2) to 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3). New cstate: current_term: 3 leader_uuid: "6151fcb16d6d4d01b2ea65767bda6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:13.235275 22304 raft_consensus.cc:1261] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:13.236344 22501 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:13.248557 22230 raft_consensus.cc:1261] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:13.249598 22501 consensus_queue.cc:1035] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:13.262784 22499 raft_consensus.cc:979] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: : Instructing follower eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:05:13.263216 22500 raft_consensus.cc:1067] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Signalling peer eef3139068244521a464f0100bc1d29c to start an election
I20240502 14:05:13.264333 22156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654"
dest_uuid: "eef3139068244521a464f0100bc1d29c"
from {username='slave'} at 127.0.0.1:44526
I20240502 14:05:13.264686 22156 raft_consensus.cc:482] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:13.264889 22156 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:13.268640 22156 raft_consensus.cc:504] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:13.269920 22156 leader_election.cc:288] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 3 election: Requested vote from peers 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501), 6151fcb16d6d4d01b2ea65767bda6076 (127.21.129.3:43421)
I20240502 14:05:13.270617 22230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c3a0e50105649c3b949b797345e463e"
I20240502 14:05:13.270751 22304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e656cba3bd84eb893113b23c825e654" candidate_uuid: "eef3139068244521a464f0100bc1d29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6151fcb16d6d4d01b2ea65767bda6076"
I20240502 14:05:13.271175 22230 raft_consensus.cc:3032] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:13.271286 22304 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:13.271589 22230 raft_consensus.cc:729] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c3a0e50105649c3b949b797345e463e, State: Running, Role: LEADER
I20240502 14:05:13.272342 22230 consensus_queue.cc:260] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:13.273610 22230 raft_consensus.cc:3037] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:13.277294 22304 raft_consensus.cc:2454] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 3.
I20240502 14:05:13.278349 22122 leader_election.cc:302] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6151fcb16d6d4d01b2ea65767bda6076, eef3139068244521a464f0100bc1d29c; no voters:
I20240502 14:05:13.279014 22501 raft_consensus.cc:2781] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:13.279310 22230 raft_consensus.cc:2454] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef3139068244521a464f0100bc1d29c in term 3.
I20240502 14:05:13.279436 22501 raft_consensus.cc:686] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 3 LEADER]: Becoming Leader. State: Replica: eef3139068244521a464f0100bc1d29c, State: Running, Role: LEADER
I20240502 14:05:13.280536 22501 consensus_queue.cc:237] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } }
I20240502 14:05:13.286851 22050 catalog_manager.cc:5484] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c reported cstate change: term changed from 2 to 3, leader changed from 8c3a0e50105649c3b949b797345e463e (127.21.129.2) to eef3139068244521a464f0100bc1d29c (127.21.129.1). New cstate: current_term: 3 leader_uuid: "eef3139068244521a464f0100bc1d29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:13.291981 22230 raft_consensus.cc:1261] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:13.293009 22501 consensus_queue.cc:1035] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:13.301520 22304 raft_consensus.cc:1261] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:13.302749 22502 consensus_queue.cc:1035] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:13.374186 22156 raft_consensus.cc:1261] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:13.375100 22479 consensus_queue.cc:1035] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:13.383738 22230 raft_consensus.cc:1261] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:13.385771 22479 consensus_queue.cc:1035] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:13.553567 22156 raft_consensus.cc:1261] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:13.554553 22479 consensus_queue.cc:1035] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef3139068244521a464f0100bc1d29c" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:13.562804 22230 raft_consensus.cc:1261] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Refusing update from remote peer 6151fcb16d6d4d01b2ea65767bda6076: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:13.564452 22479 consensus_queue.cc:1035] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:13.732067 22304 raft_consensus.cc:1261] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:13.733191 22501 consensus_queue.cc:1035] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6151fcb16d6d4d01b2ea65767bda6076" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:13.741695 22232 raft_consensus.cc:1261] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Refusing update from remote peer eef3139068244521a464f0100bc1d29c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:13.742842 22501 consensus_queue.cc:1035] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3a0e50105649c3b949b797345e463e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:13.780313 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:13.785092 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:13.785436 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:14.786120 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:14.790707 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:14.791069 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:15.791822 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:15.796571 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:15.796919 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:16.797585 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:16.802318 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:16.802678 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:17.803369 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:17.808220 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:17.808598 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:18.809352 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:18.814111 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:18.814467 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:19.815136 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:19.820717 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:19.821053 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:20.821763 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:20.826514 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:20.826861 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:21.959703 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:21.964617 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:21.964970 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:05:22.005074 22253 debug-util.cc:398] Leaking SignalData structure 0x7b0800134060 after lost signal to thread 22023
W20240502 14:05:22.005713 22253 debug-util.cc:398] Leaking SignalData structure 0x7b0800134740 after lost signal to thread 22087
W20240502 14:05:22.006407 22253 debug-util.cc:398] Leaking SignalData structure 0x7b08000d37c0 after lost signal to thread 22180
W20240502 14:05:22.007081 22253 debug-util.cc:398] Leaking SignalData structure 0x7b08000d34e0 after lost signal to thread 22256
W20240502 14:05:22.007794 22253 debug-util.cc:398] Leaking SignalData structure 0x7b0800118c40 after lost signal to thread 22330
I20240502 14:05:22.986801 22020 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240502 14:05:22.987586 22020 tablet_server.cc:178] TabletServer@127.21.129.1:0 shutting down...
W20240502 14:05:22.991878 22197 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:05:22.995774 22197 consensus_peers.cc:485] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e -> Peer eef3139068244521a464f0100bc1d29c (127.21.129.1:43373): Couldn't send request to peer eef3139068244521a464f0100bc1d29c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:23.016644 22020 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:23.017488 22020 tablet_replica.cc:330] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.018533 22020 raft_consensus.cc:2229] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.019121 22020 raft_consensus.cc:2258] T 172d1f7d66f54633a1423f26d55ba409 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.022076 22020 tablet_replica.cc:330] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.022778 22020 raft_consensus.cc:2229] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 4 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.023689 22020 raft_consensus.cc:2258] T 7d7d83ca7ba84da093d1d973c934eb95 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.025820 22020 tablet_replica.cc:330] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.026434 22020 raft_consensus.cc:2229] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.026892 22020 raft_consensus.cc:2258] T 848dc17f220448abb94b378fb2999425 P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.029117 22020 tablet_replica.cc:330] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.029834 22020 raft_consensus.cc:2229] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.030360 22020 raft_consensus.cc:2258] T 669279922bc24f55908b209483dd8271 P eef3139068244521a464f0100bc1d29c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.032538 22020 tablet_replica.cc:330] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.033152 22020 raft_consensus.cc:2229] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.033586 22020 raft_consensus.cc:2258] T d091765cb37b415ba7ebbde7e534576f P eef3139068244521a464f0100bc1d29c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.035516 22020 tablet_replica.cc:330] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.036129 22020 raft_consensus.cc:2229] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.036988 22020 raft_consensus.cc:2258] T 491e9d03d7da4dd3a9ab09767574a167 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.038975 22020 tablet_replica.cc:330] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.039589 22020 raft_consensus.cc:2229] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.040410 22020 raft_consensus.cc:2258] T 3e656cba3bd84eb893113b23c825e654 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.042407 22020 tablet_replica.cc:330] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.043002 22020 raft_consensus.cc:2229] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.043521 22020 raft_consensus.cc:2258] T 0f546cfa74964958816fa7b370229792 P eef3139068244521a464f0100bc1d29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.045603 22020 tablet_replica.cc:330] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c: stopping tablet replica
I20240502 14:05:23.046146 22020 raft_consensus.cc:2229] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.046636 22020 raft_consensus.cc:2258] T e786d30859f04ad89ac47caa0cb53622 P eef3139068244521a464f0100bc1d29c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.151841 22020 tablet_server.cc:194] TabletServer@127.21.129.1:0 shutdown complete.
W20240502 14:05:23.170594 22272 consensus_peers.cc:485] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 -> Peer eef3139068244521a464f0100bc1d29c (127.21.129.1:43373): Couldn't send request to peer eef3139068244521a464f0100bc1d29c. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.1:43373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:23.180231 22020 tablet_server.cc:178] TabletServer@127.21.129.2:0 shutting down...
I20240502 14:05:23.206513 22020 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:23.207551 22020 tablet_replica.cc:330] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.209173 22020 raft_consensus.cc:2229] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.210032 22020 raft_consensus.cc:2258] T 172d1f7d66f54633a1423f26d55ba409 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.212934 22020 tablet_replica.cc:330] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.213764 22020 raft_consensus.cc:2229] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.215302 22020 raft_consensus.cc:2258] T 848dc17f220448abb94b378fb2999425 P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.217614 22020 tablet_replica.cc:330] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.218452 22020 raft_consensus.cc:2229] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 4 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.219914 22020 raft_consensus.cc:2258] T 669279922bc24f55908b209483dd8271 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.222793 22020 tablet_replica.cc:330] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.223512 22020 raft_consensus.cc:2229] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Raft consensus shutting down.
W20240502 14:05:23.224222 22272 consensus_peers.cc:485] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 -> Peer eef3139068244521a464f0100bc1d29c (127.21.129.1:43373): Couldn't send request to peer eef3139068244521a464f0100bc1d29c. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.1:43373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:23.224248 22020 raft_consensus.cc:2258] T 7d7d83ca7ba84da093d1d973c934eb95 P 8c3a0e50105649c3b949b797345e463e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.226831 22020 tablet_replica.cc:330] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.227504 22020 raft_consensus.cc:2229] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.228065 22020 raft_consensus.cc:2258] T d091765cb37b415ba7ebbde7e534576f P 8c3a0e50105649c3b949b797345e463e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.230736 22020 tablet_replica.cc:330] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.231413 22020 raft_consensus.cc:2229] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.232092 22020 raft_consensus.cc:2258] T 491e9d03d7da4dd3a9ab09767574a167 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 14:05:23.232245 22272 consensus_peers.cc:485] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 -> Peer eef3139068244521a464f0100bc1d29c (127.21.129.1:43373): Couldn't send request to peer eef3139068244521a464f0100bc1d29c. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.1:43373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:23.234802 22020 tablet_replica.cc:330] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.235487 22020 raft_consensus.cc:2229] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.236603 22020 raft_consensus.cc:2258] T e786d30859f04ad89ac47caa0cb53622 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.238844 22020 tablet_replica.cc:330] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.239532 22020 raft_consensus.cc:2229] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.240114 22020 raft_consensus.cc:2258] T 0f546cfa74964958816fa7b370229792 P 8c3a0e50105649c3b949b797345e463e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.242512 22020 tablet_replica.cc:330] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e: stopping tablet replica
I20240502 14:05:23.243093 22020 raft_consensus.cc:2229] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.243701 22020 raft_consensus.cc:2258] T 3e656cba3bd84eb893113b23c825e654 P 8c3a0e50105649c3b949b797345e463e [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 14:05:23.295593 22270 consensus_peers.cc:485] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 -> Peer 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501): Couldn't send request to peer 8c3a0e50105649c3b949b797345e463e. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.2:37501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:23.502354 22270 consensus_peers.cc:485] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 -> Peer 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501): Couldn't send request to peer 8c3a0e50105649c3b949b797345e463e. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.2:37501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:23.653694 22270 consensus_peers.cc:485] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 -> Peer 8c3a0e50105649c3b949b797345e463e (127.21.129.2:37501): Couldn't send request to peer 8c3a0e50105649c3b949b797345e463e. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.2:37501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:23.940680 22020 tablet_server.cc:194] TabletServer@127.21.129.2:0 shutdown complete.
I20240502 14:05:23.957866 22020 tablet_server.cc:178] TabletServer@127.21.129.3:0 shutting down...
I20240502 14:05:23.979267 22020 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:23.979835 22020 tablet_replica.cc:330] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.980474 22020 raft_consensus.cc:2229] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.980978 22020 raft_consensus.cc:2258] T e786d30859f04ad89ac47caa0cb53622 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.982600 22020 tablet_replica.cc:330] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.983037 22020 raft_consensus.cc:2229] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.983539 22020 raft_consensus.cc:2258] T 7d7d83ca7ba84da093d1d973c934eb95 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.984987 22020 tablet_replica.cc:330] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.985435 22020 raft_consensus.cc:2229] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.986095 22020 raft_consensus.cc:2258] T 172d1f7d66f54633a1423f26d55ba409 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.987625 22020 tablet_replica.cc:330] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.988049 22020 raft_consensus.cc:2229] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.988476 22020 raft_consensus.cc:2258] T 3e656cba3bd84eb893113b23c825e654 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.989897 22020 tablet_replica.cc:330] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.990336 22020 raft_consensus.cc:2229] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.991012 22020 raft_consensus.cc:2258] T 0f546cfa74964958816fa7b370229792 P 6151fcb16d6d4d01b2ea65767bda6076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.992516 22020 tablet_replica.cc:330] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.992961 22020 raft_consensus.cc:2229] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.993353 22020 raft_consensus.cc:2258] T 848dc17f220448abb94b378fb2999425 P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.994858 22020 tablet_replica.cc:330] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.995332 22020 raft_consensus.cc:2229] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:23.995730 22020 raft_consensus.cc:2258] T 669279922bc24f55908b209483dd8271 P 6151fcb16d6d4d01b2ea65767bda6076 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.997162 22020 tablet_replica.cc:330] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:23.997574 22020 raft_consensus.cc:2229] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:05:23.998214 22020 raft_consensus.cc:2258] T d091765cb37b415ba7ebbde7e534576f P 6151fcb16d6d4d01b2ea65767bda6076 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:23.999612 22020 tablet_replica.cc:330] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076: stopping tablet replica
I20240502 14:05:24.000011 22020 raft_consensus.cc:2229] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:24.000432 22020 raft_consensus.cc:2258] T 491e9d03d7da4dd3a9ab09767574a167 P 6151fcb16d6d4d01b2ea65767bda6076 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:24.041975 22020 tablet_server.cc:194] TabletServer@127.21.129.3:0 shutdown complete.
I20240502 14:05:24.059123 22020 master.cc:537] Master@127.21.129.62:34865 shutting down...
I20240502 14:05:24.128111 22108 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20240502 14:05:24.178527 22020 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:24.179096 22020 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:24.179425 22020 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f4c5170f2004240ba6da1c988297f20: stopping tablet replica
I20240502 14:05:24.196564 22020 master.cc:552] Master@127.21.129.62:34865 shutdown complete.
6151fcb16d6d4d01b2ea65767bda6076 1
8c3a0e50105649c3b949b797345e463e 4
eef3139068244521a464f0100bc1d29c 4
6151fcb16d6d4d01b2ea65767bda6076 3
8c3a0e50105649c3b949b797345e463e 3
eef3139068244521a464f0100bc1d29c 3
6151fcb16d6d4d01b2ea65767bda6076 1
8c3a0e50105649c3b949b797345e463e 8
eef3139068244521a464f0100bc1d29c 0
6151fcb16d6d4d01b2ea65767bda6076 3
8c3a0e50105649c3b949b797345e463e 3
eef3139068244521a464f0100bc1d29c 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31215 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240502 14:05:24.241058 22020 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.129.62:44415
I20240502 14:05:24.242058 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:24.247177 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.247908 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.248692 22544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:24.249112 22020 server_base.cc:1034] running on GCE node
I20240502 14:05:24.249992 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:24.250173 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:24.250303 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658724250291 us; error 0 us; skew 500 ppm
I20240502 14:05:24.250803 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:24.253113 22020 webserver.cc:460] Webserver started at http://127.21.129.62:42735/ using document root <none> and password file <none>
I20240502 14:05:24.253549 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:24.253706 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:24.253948 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:24.255016 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/master-0-root/instance:
uuid: "29f24112f9fd41b5b792e51518bc64cf"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.259094 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240502 14:05:24.262020 22549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.262670 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:05:24.262918 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/master-0-root
uuid: "29f24112f9fd41b5b792e51518bc64cf"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.263154 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:24.278618 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:24.279717 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:24.314225 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.62:44415
I20240502 14:05:24.314317 22600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.62:44415 every 8 connection(s)
I20240502 14:05:24.317869 22601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.326576 22601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf: Bootstrap starting.
I20240502 14:05:24.330538 22601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:24.334211 22601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf: No bootstrap required, opened a new log
I20240502 14:05:24.336045 22601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f24112f9fd41b5b792e51518bc64cf" member_type: VOTER }
I20240502 14:05:24.336462 22601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:24.336638 22601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f24112f9fd41b5b792e51518bc64cf, State: Initialized, Role: FOLLOWER
I20240502 14:05:24.337072 22601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f24112f9fd41b5b792e51518bc64cf" member_type: VOTER }
I20240502 14:05:24.337466 22601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:24.337649 22601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:24.337860 22601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:24.341169 22601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f24112f9fd41b5b792e51518bc64cf" member_type: VOTER }
I20240502 14:05:24.341596 22601 leader_election.cc:302] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f24112f9fd41b5b792e51518bc64cf; no voters:
I20240502 14:05:24.342510 22601 leader_election.cc:288] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:24.342763 22604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:24.343955 22604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 1 LEADER]: Becoming Leader. State: Replica: 29f24112f9fd41b5b792e51518bc64cf, State: Running, Role: LEADER
I20240502 14:05:24.344597 22604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f24112f9fd41b5b792e51518bc64cf" member_type: VOTER }
I20240502 14:05:24.345172 22601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:24.349056 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f24112f9fd41b5b792e51518bc64cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f24112f9fd41b5b792e51518bc64cf" member_type: VOTER } }
I20240502 14:05:24.349366 22606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f24112f9fd41b5b792e51518bc64cf. Latest consensus state: current_term: 1 leader_uuid: "29f24112f9fd41b5b792e51518bc64cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f24112f9fd41b5b792e51518bc64cf" member_type: VOTER } }
I20240502 14:05:24.349709 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:24.350082 22606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:24.356137 22611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:24.360553 22611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:24.361490 22020 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:05:24.368968 22611 catalog_manager.cc:1347] Generated new cluster ID: 3f933634fa0d4859a8e10db1c6a79b7e
I20240502 14:05:24.369227 22611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:24.392549 22611 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:24.393621 22611 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:24.409282 22611 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf: Generated new TSK 0
I20240502 14:05:24.409785 22611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:24.427179 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:24.432650 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.433979 22623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.434499 22625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:24.435079 22020 server_base.cc:1034] running on GCE node
I20240502 14:05:24.435920 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:24.436100 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:24.436247 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658724436230 us; error 0 us; skew 500 ppm
I20240502 14:05:24.436738 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:24.438972 22020 webserver.cc:460] Webserver started at http://127.21.129.1:42231/ using document root <none> and password file <none>
I20240502 14:05:24.439433 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:24.439600 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:24.439821 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:24.440833 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-0-root/instance:
uuid: "653f39b212b2490ebb533ea039543f80"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.444808 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:05:24.447908 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.448542 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:05:24.448791 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-0-root
uuid: "653f39b212b2490ebb533ea039543f80"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.449047 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:24.468354 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:24.469380 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:24.470697 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:24.472719 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:24.472906 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.473111 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:24.473300 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.510039 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.1:46409
I20240502 14:05:24.510144 22693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.1:46409 every 8 connection(s)
I20240502 14:05:24.514480 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:24.521929 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.523257 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:24.526080 22020 server_base.cc:1034] running on GCE node
W20240502 14:05:24.526767 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:24.527171 22694 heartbeater.cc:347] Connected to a master server at 127.21.129.62:44415
I20240502 14:05:24.527626 22694 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:24.527968 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:24.528235 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:24.528431 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658724528411 us; error 0 us; skew 500 ppm
I20240502 14:05:24.528478 22694 heartbeater.cc:511] Master 127.21.129.62:44415 requested a full tablet report, sending...
I20240502 14:05:24.529227 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:24.530550 22566 ts_manager.cc:194] Registered new tserver with Master: 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409)
I20240502 14:05:24.531677 22020 webserver.cc:460] Webserver started at http://127.21.129.2:36585/ using document root <none> and password file <none>
I20240502 14:05:24.532189 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:24.532377 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:24.532454 22566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54908
I20240502 14:05:24.532631 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:24.533870 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-1-root/instance:
uuid: "c95003c717f643558d1e9d072a91400e"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.538697 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20240502 14:05:24.541918 22706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.542558 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:05:24.542807 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-1-root
uuid: "c95003c717f643558d1e9d072a91400e"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.543067 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:24.569482 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:24.570528 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:24.571822 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:24.573798 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:24.573989 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.574198 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:24.574344 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.610800 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.2:38639
I20240502 14:05:24.610922 22768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.2:38639 every 8 connection(s)
I20240502 14:05:24.615128 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:24.621310 22772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.623355 22773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:24.624955 22775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:24.625574 22020 server_base.cc:1034] running on GCE node
I20240502 14:05:24.626046 22769 heartbeater.cc:347] Connected to a master server at 127.21.129.62:44415
I20240502 14:05:24.626361 22769 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:24.626384 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:24.626686 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:24.626838 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658724626822 us; error 0 us; skew 500 ppm
I20240502 14:05:24.627120 22769 heartbeater.cc:511] Master 127.21.129.62:44415 requested a full tablet report, sending...
I20240502 14:05:24.627465 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:24.629051 22566 ts_manager.cc:194] Registered new tserver with Master: c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:24.629861 22020 webserver.cc:460] Webserver started at http://127.21.129.3:42559/ using document root <none> and password file <none>
I20240502 14:05:24.630332 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:24.630491 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:24.630438 22566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54924
I20240502 14:05:24.630820 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:24.632042 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-2-root/instance:
uuid: "ec11102d790b4644a68db70d3e31e29d"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.636129 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 14:05:24.639176 22780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.639896 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:05:24.640151 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-2-root
uuid: "ec11102d790b4644a68db70d3e31e29d"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-bdl7"
I20240502 14:05:24.640391 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714658692869395-22020-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:24.656966 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:24.657933 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:24.659224 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:24.661163 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:24.661334 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.661543 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:24.661679 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.697561 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.3:34719
I20240502 14:05:24.697647 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.3:34719 every 8 connection(s)
I20240502 14:05:24.709544 22844 heartbeater.cc:347] Connected to a master server at 127.21.129.62:44415
I20240502 14:05:24.709872 22844 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:24.710572 22844 heartbeater.cc:511] Master 127.21.129.62:44415 requested a full tablet report, sending...
I20240502 14:05:24.712219 22566 ts_manager.cc:194] Registered new tserver with Master: ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:24.712373 22020 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011970276s
I20240502 14:05:24.713577 22566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54926
I20240502 14:05:24.714674 22020 test_util.cc:268] Using random seed: 405969054
I20240502 14:05:24.734297 22566 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54928:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:05:24.736976 22566 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:05:24.799075 22808 tablet_service.cc:1467] Processing CreateTablet for tablet ee0700467d454fadac8e1aea250bda7c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 14:05:24.800443 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0700467d454fadac8e1aea250bda7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.803695 22807 tablet_service.cc:1467] Processing CreateTablet for tablet 89cd480bdc5d47c985e46046cbde365a (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 14:05:24.804980 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89cd480bdc5d47c985e46046cbde365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.812331 22733 tablet_service.cc:1467] Processing CreateTablet for tablet 89cd480bdc5d47c985e46046cbde365a (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 14:05:24.813618 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89cd480bdc5d47c985e46046cbde365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.814618 22726 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfc9ac35956473bb83fbee4a2fe215e (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 14:05:24.815874 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfc9ac35956473bb83fbee4a2fe215e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.816061 22732 tablet_service.cc:1467] Processing CreateTablet for tablet ed6e932003b640b0bc213d24330c7223 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 14:05:24.816643 22727 tablet_service.cc:1467] Processing CreateTablet for tablet 08463f4f86a945ed8169a8a468f0fb9a (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 14:05:24.817282 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6e932003b640b0bc213d24330c7223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.818750 22728 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab9d3139e124795aef9a4a59525495c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 14:05:24.819980 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9d3139e124795aef9a4a59525495c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.820380 22806 tablet_service.cc:1467] Processing CreateTablet for tablet ed6e932003b640b0bc213d24330c7223 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 14:05:24.821593 22806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6e932003b640b0bc213d24330c7223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.822692 22658 tablet_service.cc:1467] Processing CreateTablet for tablet 89cd480bdc5d47c985e46046cbde365a (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 14:05:24.823923 22652 tablet_service.cc:1467] Processing CreateTablet for tablet 08463f4f86a945ed8169a8a468f0fb9a (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 14:05:24.825093 22657 tablet_service.cc:1467] Processing CreateTablet for tablet ed6e932003b640b0bc213d24330c7223 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 14:05:24.825464 22804 tablet_service.cc:1467] Processing CreateTablet for tablet 502c7cfc908442e7a13a2a48ec4a6798 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 14:05:24.825596 22730 tablet_service.cc:1467] Processing CreateTablet for tablet 502c7cfc908442e7a13a2a48ec4a6798 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 14:05:24.826810 22651 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfc9ac35956473bb83fbee4a2fe215e (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 14:05:24.827989 22801 tablet_service.cc:1467] Processing CreateTablet for tablet 08463f4f86a945ed8169a8a468f0fb9a (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 14:05:24.829371 22802 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab9d3139e124795aef9a4a59525495c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 14:05:24.829602 22734 tablet_service.cc:1467] Processing CreateTablet for tablet ee0700467d454fadac8e1aea250bda7c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 14:05:24.830016 22650 tablet_service.cc:1467] Processing CreateTablet for tablet 03735110272046a8a958501c01b845ad (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 14:05:24.830654 22731 tablet_service.cc:1467] Processing CreateTablet for tablet 6f691ef6e0d546af865dbf2a186e5745 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 14:05:24.824349 22805 tablet_service.cc:1467] Processing CreateTablet for tablet 6f691ef6e0d546af865dbf2a186e5745 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 14:05:24.826746 22804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502c7cfc908442e7a13a2a48ec4a6798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.826737 22730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502c7cfc908442e7a13a2a48ec4a6798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.830862 22803 tablet_service.cc:1467] Processing CreateTablet for tablet ae858579e7384cb392f6d655d83c6d2c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 14:05:24.834606 22653 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab9d3139e124795aef9a4a59525495c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 14:05:24.824349 22729 tablet_service.cc:1467] Processing CreateTablet for tablet ae858579e7384cb392f6d655d83c6d2c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 14:05:24.823948 22658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89cd480bdc5d47c985e46046cbde365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.837908 22659 tablet_service.cc:1467] Processing CreateTablet for tablet ee0700467d454fadac8e1aea250bda7c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 14:05:24.838102 22654 tablet_service.cc:1467] Processing CreateTablet for tablet ae858579e7384cb392f6d655d83c6d2c (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 14:05:24.838761 22725 tablet_service.cc:1467] Processing CreateTablet for tablet 03735110272046a8a958501c01b845ad (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 14:05:24.826812 22656 tablet_service.cc:1467] Processing CreateTablet for tablet 6f691ef6e0d546af865dbf2a186e5745 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 14:05:24.842598 22659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0700467d454fadac8e1aea250bda7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.830080 22800 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfc9ac35956473bb83fbee4a2fe215e (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 14:05:24.843253 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08463f4f86a945ed8169a8a468f0fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.832872 22799 tablet_service.cc:1467] Processing CreateTablet for tablet 03735110272046a8a958501c01b845ad (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 14:05:24.843912 22651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfc9ac35956473bb83fbee4a2fe215e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.831254 22655 tablet_service.cc:1467] Processing CreateTablet for tablet 502c7cfc908442e7a13a2a48ec4a6798 (DEFAULT_TABLE table=test-workload [id=7bd1dc3fe84643dda91c4bd6e2646383]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 14:05:24.844447 22657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6e932003b640b0bc213d24330c7223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.845001 22650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03735110272046a8a958501c01b845ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.840368 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9d3139e124795aef9a4a59525495c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.845642 22654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae858579e7384cb392f6d655d83c6d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.846036 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03735110272046a8a958501c01b845ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.846661 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfc9ac35956473bb83fbee4a2fe215e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.847209 22656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f691ef6e0d546af865dbf2a186e5745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.847543 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08463f4f86a945ed8169a8a468f0fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.841317 22727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08463f4f86a945ed8169a8a468f0fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.849061 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae858579e7384cb392f6d655d83c6d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.850028 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9d3139e124795aef9a4a59525495c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.853067 22655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502c7cfc908442e7a13a2a48ec4a6798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.853116 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0700467d454fadac8e1aea250bda7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.853137 22805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f691ef6e0d546af865dbf2a186e5745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.859736 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03735110272046a8a958501c01b845ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.863561 22731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f691ef6e0d546af865dbf2a186e5745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.875525 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae858579e7384cb392f6d655d83c6d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.926466 22860 tablet_bootstrap.cc:492] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.001400 22861 tablet_bootstrap.cc:492] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.007186 22861 tablet_bootstrap.cc:654] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.012130 22859 tablet_bootstrap.cc:492] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.017947 22859 tablet_bootstrap.cc:654] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.026463 22861 tablet_bootstrap.cc:492] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.026998 22861 ts_tablet_manager.cc:1367] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240502 14:05:25.028003 22860 tablet_bootstrap.cc:654] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.029806 22861 raft_consensus.cc:348] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.030650 22861 raft_consensus.cc:374] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.030998 22861 raft_consensus.cc:729] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.031737 22861 consensus_queue.cc:260] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.034443 22694 heartbeater.cc:503] Master 127.21.129.62:44415 was elected leader, sending a full tablet report...
I20240502 14:05:25.035519 22861 ts_tablet_manager.cc:1398] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240502 14:05:25.036360 22861 tablet_bootstrap.cc:492] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.036993 22859 tablet_bootstrap.cc:492] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.037431 22859 ts_tablet_manager.cc:1367] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.010s
I20240502 14:05:25.039229 22860 tablet_bootstrap.cc:492] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.039700 22860 ts_tablet_manager.cc:1367] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.113s user 0.010s sys 0.000s
I20240502 14:05:25.041942 22861 tablet_bootstrap.cc:654] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.042807 22860 raft_consensus.cc:348] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.043382 22859 raft_consensus.cc:348] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.044226 22859 raft_consensus.cc:374] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.044606 22859 raft_consensus.cc:729] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.045301 22859 consensus_queue.cc:260] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.043874 22860 raft_consensus.cc:374] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.046955 22860 raft_consensus.cc:729] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.047129 22861 tablet_bootstrap.cc:492] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.047722 22861 ts_tablet_manager.cc:1367] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20240502 14:05:25.048234 22844 heartbeater.cc:503] Master 127.21.129.62:44415 was elected leader, sending a full tablet report...
I20240502 14:05:25.047927 22860 consensus_queue.cc:260] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.050565 22859 ts_tablet_manager.cc:1398] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240502 14:05:25.050247 22861 raft_consensus.cc:348] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.050953 22861 raft_consensus.cc:374] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.051276 22861 raft_consensus.cc:729] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.051519 22859 tablet_bootstrap.cc:492] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.051930 22861 consensus_queue.cc:260] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.054051 22861 ts_tablet_manager.cc:1398] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:05:25.055119 22861 tablet_bootstrap.cc:492] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.058092 22769 heartbeater.cc:503] Master 127.21.129.62:44415 was elected leader, sending a full tablet report...
I20240502 14:05:25.059036 22860 ts_tablet_manager.cc:1398] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.019s user 0.005s sys 0.003s
I20240502 14:05:25.059926 22860 tablet_bootstrap.cc:492] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.060799 22861 tablet_bootstrap.cc:654] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.068187 22859 tablet_bootstrap.cc:654] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.072719 22860 tablet_bootstrap.cc:654] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.077533 22859 tablet_bootstrap.cc:492] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.078153 22859 ts_tablet_manager.cc:1367] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20240502 14:05:25.078398 22861 tablet_bootstrap.cc:492] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.078831 22861 ts_tablet_manager.cc:1367] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.007s
I20240502 14:05:25.081043 22859 raft_consensus.cc:348] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.081470 22861 raft_consensus.cc:348] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.082954 22861 raft_consensus.cc:374] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.083434 22861 raft_consensus.cc:729] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.084198 22860 tablet_bootstrap.cc:492] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.084015 22861 consensus_queue.cc:260] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.082945 22859 raft_consensus.cc:374] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.084647 22860 ts_tablet_manager.cc:1367] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.010s
I20240502 14:05:25.086601 22861 ts_tablet_manager.cc:1398] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240502 14:05:25.087532 22861 tablet_bootstrap.cc:492] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.087183 22860 raft_consensus.cc:348] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.087911 22860 raft_consensus.cc:374] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.088222 22860 raft_consensus.cc:729] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.088945 22860 consensus_queue.cc:260] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.091166 22860 ts_tablet_manager.cc:1398] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240502 14:05:25.091962 22860 tablet_bootstrap.cc:492] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.093217 22861 tablet_bootstrap.cc:654] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.084823 22859 raft_consensus.cc:729] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.095877 22859 consensus_queue.cc:260] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.096717 22860 tablet_bootstrap.cc:654] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.097895 22859 ts_tablet_manager.cc:1398] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20240502 14:05:25.098759 22859 tablet_bootstrap.cc:492] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.103542 22859 tablet_bootstrap.cc:654] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.109596 22860 tablet_bootstrap.cc:492] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.109920 22860 ts_tablet_manager.cc:1367] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.015s
I20240502 14:05:25.111676 22860 raft_consensus.cc:348] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.112165 22860 raft_consensus.cc:374] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.112388 22860 raft_consensus.cc:729] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.112941 22860 consensus_queue.cc:260] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.113677 22865 raft_consensus.cc:482] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.114285 22861 tablet_bootstrap.cc:492] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.114104 22865 raft_consensus.cc:504] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.114719 22861 ts_tablet_manager.cc:1367] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.003s
I20240502 14:05:25.114976 22859 tablet_bootstrap.cc:492] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.115531 22859 ts_tablet_manager.cc:1367] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20240502 14:05:25.117414 22861 raft_consensus.cc:348] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.118113 22861 raft_consensus.cc:374] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.118484 22861 raft_consensus.cc:729] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.117946 22859 raft_consensus.cc:348] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.119047 22859 raft_consensus.cc:374] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.119407 22859 raft_consensus.cc:729] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.119172 22861 consensus_queue.cc:260] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.120484 22860 ts_tablet_manager.cc:1398] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20240502 14:05:25.120312 22859 consensus_queue.cc:260] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.122812 22860 tablet_bootstrap.cc:492] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.128242 22865 leader_election.cc:288] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.130263 22860 tablet_bootstrap.cc:654] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.130363 22859 ts_tablet_manager.cc:1398] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 14:05:25.132778 22859 tablet_bootstrap.cc:492] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.133633 22878 raft_consensus.cc:482] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.134063 22878 raft_consensus.cc:504] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.138952 22859 tablet_bootstrap.cc:654] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.144518 22878 leader_election.cc:288] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.144773 22861 ts_tablet_manager.cc:1398] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.030s user 0.006s sys 0.022s
I20240502 14:05:25.145704 22861 tablet_bootstrap.cc:492] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.145810 22867 raft_consensus.cc:482] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.146376 22867 raft_consensus.cc:504] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.148763 22867 leader_election.cc:288] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409)
I20240502 14:05:25.149633 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cd480bdc5d47c985e46046cbde365a" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.150375 22817 raft_consensus.cc:2454] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.151561 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.151587 22635 leader_election.cc:302] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.152209 22861 tablet_bootstrap.cc:654] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.152241 22818 raft_consensus.cc:2454] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.152547 22878 raft_consensus.cc:2781] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.153393 22635 leader_election.cc:302] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.154254 22876 raft_consensus.cc:2781] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.154624 22876 raft_consensus.cc:482] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.154940 22876 raft_consensus.cc:3037] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.155249 22878 raft_consensus.cc:482] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.159358 22878 raft_consensus.cc:3037] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.165685 22860 tablet_bootstrap.cc:492] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.166133 22860 ts_tablet_manager.cc:1367] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.001s
I20240502 14:05:25.166397 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cd480bdc5d47c985e46046cbde365a" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.166501 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
W20240502 14:05:25.168475 22633 leader_election.cc:341] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.169026 22860 raft_consensus.cc:348] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
W20240502 14:05:25.169442 22633 leader_election.cc:341] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.169718 22860 raft_consensus.cc:374] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.170046 22860 raft_consensus.cc:729] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.173938 22878 raft_consensus.cc:504] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.176470 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cd480bdc5d47c985e46046cbde365a" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.178402 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cd480bdc5d47c985e46046cbde365a" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.178959 22818 raft_consensus.cc:3037] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.181540 22876 raft_consensus.cc:504] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.185081 22818 raft_consensus.cc:2454] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.185346 22860 consensus_queue.cc:260] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.186064 22635 leader_election.cc:302] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.186823 22865 raft_consensus.cc:2781] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.186085 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.188513 22669 raft_consensus.cc:2379] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec11102d790b4644a68db70d3e31e29d in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.189384 22743 raft_consensus.cc:3037] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.193694 22861 tablet_bootstrap.cc:492] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.194198 22861 ts_tablet_manager.cc:1367] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.049s user 0.028s sys 0.015s
I20240502 14:05:25.194840 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
W20240502 14:05:25.197202 22782 leader_election.cc:341] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.196884 22861 raft_consensus.cc:348] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.197762 22861 raft_consensus.cc:374] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.198081 22861 raft_consensus.cc:729] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.198226 22743 raft_consensus.cc:2454] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.198797 22861 consensus_queue.cc:260] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.200359 22859 tablet_bootstrap.cc:492] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.200888 22859 ts_tablet_manager.cc:1367] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.012s
I20240502 14:05:25.199345 22784 leader_election.cc:302] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec11102d790b4644a68db70d3e31e29d; no voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.204175 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.203253 22876 leader_election.cc:288] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
W20240502 14:05:25.205708 22633 leader_election.cc:341] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.205868 22859 raft_consensus.cc:348] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.193641 22860 ts_tablet_manager.cc:1398] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.027s user 0.005s sys 0.002s
I20240502 14:05:25.198540 22878 leader_election.cc:288] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.206620 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.204145 22867 raft_consensus.cc:3037] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.207360 22861 ts_tablet_manager.cc:1398] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20240502 14:05:25.208110 22861 tablet_bootstrap.cc:492] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.209087 22860 tablet_bootstrap.cc:492] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.206595 22859 raft_consensus.cc:374] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.211406 22859 raft_consensus.cc:729] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.212016 22861 tablet_bootstrap.cc:654] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.212298 22859 consensus_queue.cc:260] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.213382 22867 raft_consensus.cc:2726] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.214676 22860 tablet_bootstrap.cc:654] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.206601 22865 raft_consensus.cc:686] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.219030 22865 consensus_queue.cc:237] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.227445 22818 raft_consensus.cc:2454] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.231002 22635 leader_election.cc:302] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters: c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.232174 22878 raft_consensus.cc:2781] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.232647 22878 raft_consensus.cc:686] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.233408 22878 consensus_queue.cc:237] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.262005 22566 catalog_manager.cc:5484] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:25.269161 22859 ts_tablet_manager.cc:1398] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.065s user 0.018s sys 0.038s
I20240502 14:05:25.270198 22859 tablet_bootstrap.cc:492] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.270574 22861 tablet_bootstrap.cc:492] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.271019 22861 ts_tablet_manager.cc:1367] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.030s
I20240502 14:05:25.273643 22861 raft_consensus.cc:348] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.274348 22861 raft_consensus.cc:374] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.274652 22861 raft_consensus.cc:729] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.275338 22861 consensus_queue.cc:260] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.275817 22859 tablet_bootstrap.cc:654] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.277669 22860 tablet_bootstrap.cc:492] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.277719 22861 ts_tablet_manager.cc:1398] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 14:05:25.278353 22860 ts_tablet_manager.cc:1367] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.069s user 0.034s sys 0.023s
I20240502 14:05:25.278860 22861 tablet_bootstrap.cc:492] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.279711 22563 catalog_manager.cc:5484] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:25.281157 22860 raft_consensus.cc:348] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.281895 22860 raft_consensus.cc:374] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.282245 22860 raft_consensus.cc:729] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.282996 22860 consensus_queue.cc:260] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.285148 22860 ts_tablet_manager.cc:1398] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:25.285720 22861 tablet_bootstrap.cc:654] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.286156 22860 tablet_bootstrap.cc:492] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.287144 22859 tablet_bootstrap.cc:492] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.287627 22859 ts_tablet_manager.cc:1367] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240502 14:05:25.290377 22859 raft_consensus.cc:348] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.291110 22859 raft_consensus.cc:374] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.291415 22859 raft_consensus.cc:729] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.292130 22860 tablet_bootstrap.cc:654] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.292266 22859 consensus_queue.cc:260] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.297291 22859 ts_tablet_manager.cc:1398] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20240502 14:05:25.298164 22859 tablet_bootstrap.cc:492] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.306464 22859 tablet_bootstrap.cc:654] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.311331 22861 tablet_bootstrap.cc:492] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.311774 22861 ts_tablet_manager.cc:1367] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.016s
I20240502 14:05:25.312330 22859 tablet_bootstrap.cc:492] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.312754 22859 ts_tablet_manager.cc:1367] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240502 14:05:25.314232 22861 raft_consensus.cc:348] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.314980 22861 raft_consensus.cc:374] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.315284 22861 raft_consensus.cc:729] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.315318 22859 raft_consensus.cc:348] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.316100 22859 raft_consensus.cc:374] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.316453 22859 raft_consensus.cc:729] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.316193 22861 consensus_queue.cc:260] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.317363 22860 tablet_bootstrap.cc:492] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.317183 22859 consensus_queue.cc:260] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.317798 22860 ts_tablet_manager.cc:1367] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.014s
I20240502 14:05:25.318450 22861 ts_tablet_manager.cc:1398] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:25.319334 22861 tablet_bootstrap.cc:492] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.320336 22860 raft_consensus.cc:348] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.321033 22860 raft_consensus.cc:374] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.321309 22860 raft_consensus.cc:729] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.321765 22859 ts_tablet_manager.cc:1398] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:05:25.321997 22860 consensus_queue.cc:260] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.322649 22859 tablet_bootstrap.cc:492] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.324131 22860 ts_tablet_manager.cc:1398] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:25.324966 22861 tablet_bootstrap.cc:654] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.324993 22860 tablet_bootstrap.cc:492] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.328485 22859 tablet_bootstrap.cc:654] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.333098 22860 tablet_bootstrap.cc:654] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.334210 22865 raft_consensus.cc:482] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.334640 22865 raft_consensus.cc:504] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.336500 22865 leader_election.cc:288] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.338164 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.338838 22743 raft_consensus.cc:2454] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.339341 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.340024 22633 leader_election.cc:302] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
W20240502 14:05:25.340730 22635 leader_election.cc:341] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.340898 22865 raft_consensus.cc:2781] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.341315 22865 raft_consensus.cc:482] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.341641 22865 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.346199 22860 tablet_bootstrap.cc:492] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.346623 22860 ts_tablet_manager.cc:1367] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.005s
I20240502 14:05:25.348734 22865 raft_consensus.cc:504] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.349146 22860 raft_consensus.cc:348] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.349985 22860 raft_consensus.cc:374] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.350359 22860 raft_consensus.cc:729] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.350472 22865 leader_election.cc:288] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.351338 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.351099 22860 consensus_queue.cc:260] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.351840 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.351940 22861 tablet_bootstrap.cc:492] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.351861 22743 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:25.353082 22635 leader_election.cc:341] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.353868 22859 tablet_bootstrap.cc:492] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.354350 22859 ts_tablet_manager.cc:1367] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.008s
I20240502 14:05:25.356050 22860 ts_tablet_manager.cc:1398] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:05:25.356918 22860 tablet_bootstrap.cc:492] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.356948 22859 raft_consensus.cc:348] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.357743 22859 raft_consensus.cc:374] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.358024 22859 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.358299 22743 raft_consensus.cc:2454] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.359102 22861 ts_tablet_manager.cc:1367] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.007s
I20240502 14:05:25.358647 22859 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.361315 22633 leader_election.cc:302] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters: ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.361646 22861 raft_consensus.cc:348] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.362282 22861 raft_consensus.cc:374] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.362542 22861 raft_consensus.cc:729] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.362614 22860 tablet_bootstrap.cc:654] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.362733 22859 ts_tablet_manager.cc:1398] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:05:25.363679 22859 tablet_bootstrap.cc:492] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.363281 22861 consensus_queue.cc:260] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.364286 22878 raft_consensus.cc:2781] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.364769 22878 raft_consensus.cc:686] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.365533 22878 consensus_queue.cc:237] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
W20240502 14:05:25.370128 22614 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:05:25.373871 22861 ts_tablet_manager.cc:1398] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20240502 14:05:25.375778 22859 tablet_bootstrap.cc:654] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.376076 22861 tablet_bootstrap.cc:492] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.378635 22868 raft_consensus.cc:482] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.379102 22868 raft_consensus.cc:504] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.380885 22860 tablet_bootstrap.cc:492] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.381218 22861 tablet_bootstrap.cc:654] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.381320 22860 ts_tablet_manager.cc:1367] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240502 14:05:25.381335 22868 leader_election.cc:288] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.384070 22860 raft_consensus.cc:348] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.384850 22860 raft_consensus.cc:374] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.385164 22860 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.385879 22860 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.394709 22861 tablet_bootstrap.cc:492] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.395102 22861 ts_tablet_manager.cc:1367] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.003s
I20240502 14:05:25.395763 22860 ts_tablet_manager.cc:1398] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20240502 14:05:25.396646 22859 tablet_bootstrap.cc:492] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.389624 22564 catalog_manager.cc:5484] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:25.397121 22859 ts_tablet_manager.cc:1367] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.010s
I20240502 14:05:25.397342 22861 raft_consensus.cc:348] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.397986 22861 raft_consensus.cc:374] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.398675 22861 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.398409 22886 raft_consensus.cc:482] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.399296 22861 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.399269 22886 raft_consensus.cc:504] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.399999 22859 raft_consensus.cc:348] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.401044 22859 raft_consensus.cc:374] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.401336 22859 raft_consensus.cc:729] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.401412 22861 ts_tablet_manager.cc:1398] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:05:25.401474 22886 leader_election.cc:288] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.402321 22861 tablet_bootstrap.cc:492] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: Bootstrap starting.
I20240502 14:05:25.402114 22859 consensus_queue.cc:260] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.403316 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.403898 22743 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 0.
I20240502 14:05:25.405153 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.405755 22669 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 0.
I20240502 14:05:25.406196 22859 ts_tablet_manager.cc:1398] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20240502 14:05:25.407040 22859 tablet_bootstrap.cc:492] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.407348 22861 tablet_bootstrap.cc:654] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.407528 22782 leader_election.cc:302] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.409130 22886 raft_consensus.cc:2781] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.409471 22886 raft_consensus.cc:482] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.409783 22886 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.412497 22859 tablet_bootstrap.cc:654] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.416536 22886 raft_consensus.cc:504] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.418879 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:25.419423 22669 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.423326 22867 raft_consensus.cc:482] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.423753 22867 raft_consensus.cc:504] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.425523 22867 leader_election.cc:288] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.429859 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.430511 22668 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 0.
I20240502 14:05:25.432897 22784 leader_election.cc:302] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.433835 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
W20240502 14:05:25.435048 22782 leader_election.cc:341] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.439055 22867 raft_consensus.cc:2781] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.439450 22867 raft_consensus.cc:482] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.439772 22867 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.446020 22669 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 1.
I20240502 14:05:25.446987 22784 leader_election.cc:302] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.447126 22868 raft_consensus.cc:482] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.448247 22868 raft_consensus.cc:504] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.450078 22868 leader_election.cc:288] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.450162 22878 raft_consensus.cc:482] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.450621 22878 raft_consensus.cc:504] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.452423 22878 leader_election.cc:288] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.453022 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6e932003b640b0bc213d24330c7223" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.453595 22743 raft_consensus.cc:2454] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.454634 22633 leader_election.cc:302] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:25.455394 22878 raft_consensus.cc:2781] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.455730 22878 raft_consensus.cc:482] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.456054 22878 raft_consensus.cc:3037] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.456310 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6e932003b640b0bc213d24330c7223" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.456887 22818 raft_consensus.cc:2454] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.459779 22907 raft_consensus.cc:2781] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.462185 22878 raft_consensus.cc:504] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.462832 22861 tablet_bootstrap.cc:492] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: No bootstrap required, opened a new log
I20240502 14:05:25.463328 22861 ts_tablet_manager.cc:1367] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.047s
I20240502 14:05:25.465077 22867 raft_consensus.cc:504] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.465783 22861 raft_consensus.cc:348] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.466462 22861 raft_consensus.cc:374] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.466719 22861 raft_consensus.cc:729] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.467352 22861 consensus_queue.cc:260] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.467972 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.469312 22861 ts_tablet_manager.cc:1398] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20240502 14:05:25.469276 22782 leader_election.cc:341] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.470701 22885 raft_consensus.cc:482] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.471318 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.471120 22885 raft_consensus.cc:504] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.471920 22743 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.398160 22860 tablet_bootstrap.cc:492] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.473637 22885 leader_election.cc:288] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.474252 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.474553 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.475097 22818 raft_consensus.cc:2454] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
W20240502 14:05:25.475484 22633 leader_election.cc:341] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.475960 22886 leader_election.cc:288] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.476153 22635 leader_election.cc:302] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters: c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.476436 22886 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.477123 22885 raft_consensus.cc:2781] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.477455 22885 raft_consensus.cc:482] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.477777 22885 raft_consensus.cc:3037] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.480094 22743 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 1.
I20240502 14:05:25.476832 22886 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.482594 22907 raft_consensus.cc:686] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:25.482805 22878 leader_election.cc:288] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.483763 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6e932003b640b0bc213d24330c7223" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.484318 22818 raft_consensus.cc:3037] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.487385 22868 raft_consensus.cc:482] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.487946 22867 leader_election.cc:288] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.487884 22868 raft_consensus.cc:504] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.489112 22878 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.483701 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6e932003b640b0bc213d24330c7223" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.490125 22744 raft_consensus.cc:3037] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.489769 22878 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.491364 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.492045 22742 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 0.
I20240502 14:05:25.492867 22878 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.494021 22782 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.490509 22818 raft_consensus.cc:2454] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.494493 22886 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.493930 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.498446 22635 leader_election.cc:302] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.499480 22878 raft_consensus.cc:2781] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.500691 22878 raft_consensus.cc:686] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.501655 22878 consensus_queue.cc:237] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.502507 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.495440 22865 raft_consensus.cc:482] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.500718 22860 tablet_bootstrap.cc:654] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.503451 22865 raft_consensus.cc:504] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.500865 22744 raft_consensus.cc:2454] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.508555 22865 leader_election.cc:288] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.505789 22868 leader_election.cc:288] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.505131 22741 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.510147 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.510764 22818 raft_consensus.cc:2454] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.511641 22885 raft_consensus.cc:504] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.511901 22635 leader_election.cc:302] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.512771 22865 raft_consensus.cc:2781] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.513125 22865 raft_consensus.cc:482] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.513432 22865 raft_consensus.cc:3037] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.514950 22633 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:25.497702 22886 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.515678 22817 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.516043 22876 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.516451 22876 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.516498 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.483711 22907 consensus_queue.cc:237] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.519508 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.516844 22876 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.520040 22817 raft_consensus.cc:3037] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.522236 22868 raft_consensus.cc:482] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.523087 22868 raft_consensus.cc:504] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:25.513341 22885 leader_election.cc:288] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.516916 22886 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.525342 22886 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.528340 22884 raft_consensus.cc:482] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.528815 22884 raft_consensus.cc:504] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.532188 22884 leader_election.cc:288] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.530063 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.532881 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.533103 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.533494 22743 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.520716 22744 raft_consensus.cc:2454] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 0.
I20240502 14:05:25.539748 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
W20240502 14:05:25.508841 22784 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.1:46409, user_credentials={real_user=slave}} blocked reactor thread for 61911.8us
I20240502 14:05:25.535463 22633 leader_election.cc:302] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:25.548051 22868 leader_election.cc:288] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719), 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409)
I20240502 14:05:25.553120 22886 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.556737 22886 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.558220 22878 raft_consensus.cc:2781] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.558822 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
W20240502 14:05:25.556070 22635 leader_election.cc:341] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.562233 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.559319 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.563323 22818 raft_consensus.cc:2454] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 0.
W20240502 14:05:25.566357 22633 leader_election.cc:341] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.567080 22876 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.567524 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:25.567633 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.568138 22669 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.568837 22860 tablet_bootstrap.cc:492] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.568010 22566 catalog_manager.cc:5484] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 0 to 1, leader changed from <none> to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 1 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:25.569198 22860 ts_tablet_manager.cc:1367] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.171s user 0.009s sys 0.007s
I20240502 14:05:25.571022 22860 raft_consensus.cc:348] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.571702 22860 raft_consensus.cc:374] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.571951 22860 raft_consensus.cc:729] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.572618 22860 consensus_queue.cc:260] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.559907 22878 raft_consensus.cc:482] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.573560 22878 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.578608 22860 ts_tablet_manager.cc:1398] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:05:25.579237 22860 tablet_bootstrap.cc:492] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e: Bootstrap starting.
I20240502 14:05:25.562647 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.574508 22741 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.574985 22667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.581660 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.584394 22667 raft_consensus.cc:2379] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c95003c717f643558d1e9d072a91400e in current term 1: Already voted for candidate ec11102d790b4644a68db70d3e31e29d in this term.
I20240502 14:05:25.584579 22817 raft_consensus.cc:2454] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.577492 22666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.589061 22667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:25.588873 22665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.585216 22868 raft_consensus.cc:482] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.587963 22668 raft_consensus.cc:2379] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec11102d790b4644a68db70d3e31e29d in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.588253 22816 raft_consensus.cc:2379] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c95003c717f643558d1e9d072a91400e in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.559620 22565 catalog_manager.cc:5484] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:25.593762 22868 raft_consensus.cc:504] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.597364 22878 raft_consensus.cc:504] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.599574 22878 leader_election.cc:288] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.601034 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.601646 22744 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.603778 22859 tablet_bootstrap.cc:492] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.604090 22741 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 1.
I20240502 14:05:25.604252 22859 ts_tablet_manager.cc:1367] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.197s user 0.048s sys 0.061s
I20240502 14:05:25.606103 22859 raft_consensus.cc:348] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.606536 22859 raft_consensus.cc:374] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.606778 22859 raft_consensus.cc:729] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.596902 22868 leader_election.cc:288] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.607758 22666 raft_consensus.cc:2379] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c95003c717f643558d1e9d072a91400e in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.609350 22859 consensus_queue.cc:260] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.589416 22865 raft_consensus.cc:504] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.612463 22744 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.610213 22669 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 1.
I20240502 14:05:25.613442 22665 raft_consensus.cc:2379] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c95003c717f643558d1e9d072a91400e in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.614090 22859 ts_tablet_manager.cc:1398] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:05:25.614367 22784 leader_election.cc:302] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters: c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.614882 22859 tablet_bootstrap.cc:492] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: Bootstrap starting.
I20240502 14:05:25.615572 22865 leader_election.cc:288] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
W20240502 14:05:25.616027 22710 leader_election.cc:341] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:25.616266 22886 raft_consensus.cc:2781] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election won for term 1
W20240502 14:05:25.616588 22710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:34719, user_credentials={real_user=slave}} blocked reactor thread for 51887.9us
I20240502 14:05:25.616829 22886 raft_consensus.cc:686] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:25.594271 22635 leader_election.cc:302] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters: c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.615900 22633 leader_election.cc:302] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:25.618593 22782 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters: 653f39b212b2490ebb533ea039543f80
I20240502 14:05:25.619598 22710 leader_election.cc:302] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.619956 22886 consensus_queue.cc:237] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.623919 22860 tablet_bootstrap.cc:654] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.620703 22865 raft_consensus.cc:2781] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 14:05:25.621758 22770 tablet.cc:2369] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:25.622416 22907 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.625140 22878 raft_consensus.cc:2781] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.625397 22907 raft_consensus.cc:686] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:25.625612 22878 raft_consensus.cc:686] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.624850 22876 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.625664 22865 raft_consensus.cc:686] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.626353 22878 consensus_queue.cc:237] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.626205 22907 consensus_queue.cc:237] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.627277 22865 consensus_queue.cc:237] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.623773 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.630348 22816 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.631048 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.631815 22741 raft_consensus.cc:2379] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 653f39b212b2490ebb533ea039543f80 in current term 1: Already voted for candidate ec11102d790b4644a68db70d3e31e29d in this term.
I20240502 14:05:25.632493 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.632508 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.633546 22710 leader_election.cc:302] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c95003c717f643558d1e9d072a91400e; no voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.633977 22817 raft_consensus.cc:3037] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.634661 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:25.635123 22868 raft_consensus.cc:2781] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:25.635535 22868 raft_consensus.cc:482] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:25.635926 22868 raft_consensus.cc:3037] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.620703 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:25.642899 22817 raft_consensus.cc:2454] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.643848 22710 leader_election.cc:302] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c95003c717f643558d1e9d072a91400e; no voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.643496 22635 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 653f39b212b2490ebb533ea039543f80; no voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.654295 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.655092 22669 raft_consensus.cc:2379] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c95003c717f643558d1e9d072a91400e in current term 1: Already voted for candidate ec11102d790b4644a68db70d3e31e29d in this term.
I20240502 14:05:25.654567 22563 catalog_manager.cc:5484] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:25.657351 22816 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 1.
I20240502 14:05:25.657809 22865 consensus_queue.cc:1035] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.657662 22563 catalog_manager.cc:5484] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:25.656153 22878 raft_consensus.cc:2726] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.672631 22868 raft_consensus.cc:504] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.673915 22744 raft_consensus.cc:2379] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 653f39b212b2490ebb533ea039543f80 in current term 1: Already voted for candidate c95003c717f643558d1e9d072a91400e in this term.
I20240502 14:05:25.676579 22868 leader_election.cc:288] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.662238 22635 leader_election.cc:302] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:25.680228 22912 raft_consensus.cc:3037] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.685765 22859 tablet_bootstrap.cc:654] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:25.688381 22917 raft_consensus.cc:2726] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:05:25.688467 22710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.1:46409, user_credentials={real_user=slave}} blocked reactor thread for 49617.9us
I20240502 14:05:25.689306 22710 leader_election.cc:302] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c95003c717f643558d1e9d072a91400e; no voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.694368 22912 raft_consensus.cc:2726] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.695214 22878 raft_consensus.cc:2781] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:25.697217 22917 raft_consensus.cc:2726] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.708520 22865 consensus_queue.cc:1035] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.709991 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:25.710141 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.710950 22816 raft_consensus.cc:2379] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c95003c717f643558d1e9d072a91400e in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.712488 22860 tablet_bootstrap.cc:492] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e: No bootstrap required, opened a new log
I20240502 14:05:25.712927 22860 ts_tablet_manager.cc:1367] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e: Time spent bootstrapping tablet: real 0.134s user 0.019s sys 0.019s
I20240502 14:05:25.715109 22710 leader_election.cc:302] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c95003c717f643558d1e9d072a91400e; no voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.700142 22878 raft_consensus.cc:686] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:25.716802 22878 consensus_queue.cc:237] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.718734 22918 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.720281 22860 raft_consensus.cc:348] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.720995 22860 raft_consensus.cc:374] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.721288 22860 raft_consensus.cc:729] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.721904 22860 consensus_queue.cc:260] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.723870 22860 ts_tablet_manager.cc:1398] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240502 14:05:25.724898 22918 raft_consensus.cc:2726] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.725203 22566 catalog_manager.cc:5484] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 0 to 1, leader changed from <none> to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 1 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:25.728816 22859 tablet_bootstrap.cc:492] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: No bootstrap required, opened a new log
I20240502 14:05:25.729249 22859 ts_tablet_manager.cc:1367] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: Time spent bootstrapping tablet: real 0.115s user 0.017s sys 0.006s
I20240502 14:05:25.731643 22859 raft_consensus.cc:348] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.732271 22859 raft_consensus.cc:374] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:25.732535 22859 raft_consensus.cc:729] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Initialized, Role: FOLLOWER
I20240502 14:05:25.733105 22859 consensus_queue.cc:260] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.735054 22859 ts_tablet_manager.cc:1398] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:05:25.741813 22564 catalog_manager.cc:5484] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 0 to 1, leader changed from <none> to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 1 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:25.768575 22884 consensus_queue.cc:1035] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.775836 22865 consensus_queue.cc:1035] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.777103 22710 leader_election.cc:302] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c95003c717f643558d1e9d072a91400e; no voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.777876 22918 raft_consensus.cc:2726] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.781807 22566 catalog_manager.cc:5484] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:25.827579 22907 raft_consensus.cc:482] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:25.827957 22907 raft_consensus.cc:504] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.829386 22907 leader_election.cc:288] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:25.830044 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:25.830271 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95003c717f643558d1e9d072a91400e" is_pre_election: true
I20240502 14:05:25.830937 22741 raft_consensus.cc:2379] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec11102d790b4644a68db70d3e31e29d in current term 1: Already voted for candidate 653f39b212b2490ebb533ea039543f80 in this term.
I20240502 14:05:25.831735 22782 leader_election.cc:302] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec11102d790b4644a68db70d3e31e29d; no voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.832289 22907 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:25.837102 22907 raft_consensus.cc:2726] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:25.837308 22865 consensus_queue.cc:1035] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.846952 22878 consensus_queue.cc:1035] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.882941 22816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
mode: GRACEFUL
new_leader_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:52116
I20240502 14:05:25.883443 22816 raft_consensus.cc:595] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Received request to transfer leadership to TS 653f39b212b2490ebb533ea039543f80
I20240502 14:05:25.889214 22816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
mode: GRACEFUL
new_leader_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:52116
I20240502 14:05:25.889588 22816 raft_consensus.cc:595] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Received request to transfer leadership to TS c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.899771 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:40620
I20240502 14:05:25.900188 22669 raft_consensus.cc:595] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Received request to transfer leadership to TS c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.904012 22816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
mode: GRACEFUL
new_leader_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:52116
I20240502 14:05:25.904422 22816 raft_consensus.cc:595] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Received request to transfer leadership to TS c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.908135 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:40620
I20240502 14:05:25.908543 22669 raft_consensus.cc:595] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Received request to transfer leadership to TS c95003c717f643558d1e9d072a91400e
I20240502 14:05:25.912214 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03735110272046a8a958501c01b845ad"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:40620
I20240502 14:05:25.912604 22669 raft_consensus.cc:595] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Received request to transfer leadership to TS ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:25.940057 22907 consensus_queue.cc:1035] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.944592 22878 consensus_queue.cc:1035] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.947057 22886 raft_consensus.cc:979] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: : Instructing follower c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:25.947746 22908 raft_consensus.cc:1067] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Signalling peer c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:25.952410 22743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:45422
I20240502 14:05:25.952931 22743 raft_consensus.cc:482] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:25.953286 22743 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:25.955760 22878 consensus_queue.cc:1035] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:25.957625 22906 consensus_queue.cc:1035] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:25.961148 22743 raft_consensus.cc:504] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.963445 22743 leader_election.cc:288] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:25.964016 22665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:25.967841 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:25.968454 22816 raft_consensus.cc:3032] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:25.969004 22816 raft_consensus.cc:729] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:25.969758 22816 consensus_queue.cc:260] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.971004 22816 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:25.977284 22816 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:25.978396 22710 leader_election.cc:302] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters: 653f39b212b2490ebb533ea039543f80
I20240502 14:05:25.979256 22918 raft_consensus.cc:2781] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:25.979704 22918 raft_consensus.cc:686] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Leader. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:25.980417 22918 consensus_queue.cc:237] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:25.991155 22564 catalog_manager.cc:5484] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e reported cstate change: term changed from 1 to 2, leader changed from ec11102d790b4644a68db70d3e31e29d (127.21.129.3) to c95003c717f643558d1e9d072a91400e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "c95003c717f643558d1e9d072a91400e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:26.019266 22906 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.024583 22908 raft_consensus.cc:979] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: : Instructing follower c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:26.025030 22907 raft_consensus.cc:1067] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Signalling peer c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:26.026494 22743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:45422
I20240502 14:05:26.027004 22743 raft_consensus.cc:482] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:26.027387 22743 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.030270 22741 raft_consensus.cc:3037] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:26.030423 22907 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.035923 22743 raft_consensus.cc:504] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.038446 22878 consensus_queue.cc:1035] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.041056 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:26.041503 22743 leader_election.cc:288] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:26.041644 22669 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.043316 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:26.043907 22816 raft_consensus.cc:3032] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:26.044204 22816 raft_consensus.cc:729] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:26.044946 22816 consensus_queue.cc:260] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.047020 22816 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.047261 22885 raft_consensus.cc:979] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: : Instructing follower c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:26.047694 22885 raft_consensus.cc:1067] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Signalling peer c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:26.048408 22669 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:26.049760 22710 leader_election.cc:302] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:26.050781 22920 raft_consensus.cc:2781] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:26.051270 22920 raft_consensus.cc:686] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Leader. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:26.051510 22865 consensus_queue.cc:1035] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.052052 22920 consensus_queue.cc:237] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.053067 22907 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.056159 22816 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:26.056458 22742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:45410
I20240502 14:05:26.059738 22742 raft_consensus.cc:482] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:26.060109 22742 raft_consensus.cc:3037] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.066283 22742 raft_consensus.cc:504] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.073385 22742 leader_election.cc:288] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:26.075245 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:26.075810 22669 raft_consensus.cc:3032] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:26.076278 22669 raft_consensus.cc:729] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:26.075810 22566 catalog_manager.cc:5484] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e reported cstate change: term changed from 1 to 2, leader changed from ec11102d790b4644a68db70d3e31e29d (127.21.129.3) to c95003c717f643558d1e9d072a91400e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "c95003c717f643558d1e9d072a91400e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:26.076550 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfc9ac35956473bb83fbee4a2fe215e" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:26.077088 22817 raft_consensus.cc:3037] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.076979 22669 consensus_queue.cc:260] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.077950 22906 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.078436 22669 raft_consensus.cc:3037] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.083352 22817 raft_consensus.cc:2454] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:26.086436 22669 raft_consensus.cc:2454] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:26.087646 22710 leader_election.cc:302] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:26.090198 22920 raft_consensus.cc:2781] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:26.090848 22920 raft_consensus.cc:686] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Leader. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:26.091727 22920 consensus_queue.cc:237] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.095134 22908 raft_consensus.cc:979] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: : Instructing follower 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:26.095585 22906 raft_consensus.cc:1067] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Signalling peer 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:26.097143 22665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:40604
I20240502 14:05:26.097620 22665 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:26.097887 22665 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.099733 22566 catalog_manager.cc:5484] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e reported cstate change: term changed from 1 to 2, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to c95003c717f643558d1e9d072a91400e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "c95003c717f643558d1e9d072a91400e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:26.104794 22665 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.106422 22665 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 2 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:26.107432 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:26.107991 22817 raft_consensus.cc:3032] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:26.108042 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:26.108366 22817 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:26.108618 22743 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.109136 22817 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.110352 22817 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.114529 22743 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 2.
I20240502 14:05:26.115437 22633 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:26.116037 22878 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:26.115993 22817 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 2.
I20240502 14:05:26.116608 22878 raft_consensus.cc:686] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:26.117287 22878 consensus_queue.cc:237] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.122370 22563 catalog_manager.cc:5484] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 1 to 2, leader changed from ec11102d790b4644a68db70d3e31e29d (127.21.129.3) to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 2 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:26.244529 22878 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.248499 22878 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.251622 22884 raft_consensus.cc:979] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: : Instructing follower c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:26.252041 22884 raft_consensus.cc:1067] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Signalling peer c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:26.255936 22742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:45410
I20240502 14:05:26.256443 22742 raft_consensus.cc:482] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:26.256793 22742 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.263029 22742 raft_consensus.cc:504] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.264212 22743 raft_consensus.cc:1226] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Rejecting Update request from peer 653f39b212b2490ebb533ea039543f80 for earlier term 1. Current term is 2. Ops: []
I20240502 14:05:26.264827 22742 leader_election.cc:288] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:26.266232 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:26.266888 22817 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.267635 22878 consensus_queue.cc:1046] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.268517 22885 raft_consensus.cc:3032] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:26.268746 22885 raft_consensus.cc:729] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:26.266790 22665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:26.269320 22885 consensus_queue.cc:260] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.270316 22885 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.273424 22817 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:26.274315 22710 leader_election.cc:302] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:26.274968 22920 raft_consensus.cc:2781] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:26.275401 22920 raft_consensus.cc:686] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Leader. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:26.276077 22920 consensus_queue.cc:237] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.279261 22665 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 2.
I20240502 14:05:26.281947 22563 catalog_manager.cc:5484] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e reported cstate change: term changed from 1 to 2, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to c95003c717f643558d1e9d072a91400e (127.21.129.2). New cstate: current_term: 2 leader_uuid: "c95003c717f643558d1e9d072a91400e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:26.301406 22865 consensus_queue.cc:1035] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.306339 22885 raft_consensus.cc:979] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: : Instructing follower ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:26.306799 22878 raft_consensus.cc:1067] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Signalling peer ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:26.308346 22816 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03735110272046a8a958501c01b845ad"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:52094
I20240502 14:05:26.308857 22816 raft_consensus.cc:482] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:26.309221 22816 raft_consensus.cc:3037] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.311435 22878 consensus_queue.cc:1035] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:26.315115 22816 raft_consensus.cc:504] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.318827 22816 leader_election.cc:288] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 2 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:26.320096 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:26.320286 22665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03735110272046a8a958501c01b845ad" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:26.320556 22743 raft_consensus.cc:3037] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.320927 22665 raft_consensus.cc:3032] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:26.321291 22665 raft_consensus.cc:729] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:26.322043 22665 consensus_queue.cc:260] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.322953 22665 raft_consensus.cc:3037] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.325492 22743 raft_consensus.cc:2454] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 2.
I20240502 14:05:26.326437 22782 leader_election.cc:302] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:26.326982 22665 raft_consensus.cc:2454] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 2.
I20240502 14:05:26.327210 22906 raft_consensus.cc:2781] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:26.327698 22906 raft_consensus.cc:686] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:26.328433 22906 consensus_queue.cc:237] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:26.333577 22566 catalog_manager.cc:5484] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 1 to 2, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 2 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:26.426771 22665 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:26.431411 22665 raft_consensus.cc:1261] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.432564 22920 consensus_queue.cc:1035] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.439265 22816 raft_consensus.cc:1261] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.440699 22920 consensus_queue.cc:1035] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.484504 22816 raft_consensus.cc:1261] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.484712 22665 raft_consensus.cc:1261] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.485534 22920 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.486153 22949 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.597909 22816 raft_consensus.cc:1261] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.598939 22918 consensus_queue.cc:1035] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.606014 22665 raft_consensus.cc:1261] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.607650 22949 consensus_queue.cc:1035] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.646353 22743 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.647342 22878 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.654712 22816 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.655886 22878 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.760977 22665 raft_consensus.cc:1261] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.762007 22906 consensus_queue.cc:1035] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.768782 22743 raft_consensus.cc:1261] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.770007 22951 consensus_queue.cc:1035] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.780162 22665 raft_consensus.cc:1261] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.781200 22949 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:26.788926 22817 raft_consensus.cc:1261] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:26.790163 22920 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:28.936786 22020 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240502 14:05:28.937556 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:28.958580 22665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:40624
I20240502 14:05:28.959049 22665 raft_consensus.cc:595] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Received request to transfer leadership to TS ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:28.960880 22665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:40624
I20240502 14:05:28.961283 22665 raft_consensus.cc:595] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 LEADER]: Received request to transfer leadership to TS ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:28.962204 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 14:05:28.962980 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:29.488271 22878 raft_consensus.cc:979] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: : Instructing follower ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:29.488754 22954 raft_consensus.cc:1067] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 LEADER]: Signalling peer ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:29.489928 22817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:52094
I20240502 14:05:29.490386 22817 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:29.490664 22817 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:29.494864 22817 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:29.496210 22817 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 3 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:29.496913 22665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:29.497107 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:29.497501 22665 raft_consensus.cc:3032] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:29.497625 22742 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:29.497850 22665 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:29.498591 22665 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:29.499775 22665 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:29.503710 22742 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 3.
I20240502 14:05:29.504717 22782 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:29.505343 22955 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:29.505473 22665 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 3.
I20240502 14:05:29.506285 22955 raft_consensus.cc:686] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:29.507057 22955 consensus_queue.cc:237] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:29.512758 22564 catalog_manager.cc:5484] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 2 to 3, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 3 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:29.538792 22878 raft_consensus.cc:979] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: : Instructing follower ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:29.539124 22954 raft_consensus.cc:1067] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Signalling peer ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:29.540242 22817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:52094
I20240502 14:05:29.540653 22817 raft_consensus.cc:482] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:29.540954 22817 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:29.545683 22817 raft_consensus.cc:504] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:29.547130 22817 leader_election.cc:288] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 2 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:29.548065 22665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:29.548508 22665 raft_consensus.cc:3032] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:29.548732 22665 raft_consensus.cc:729] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:29.549265 22665 consensus_queue.cc:260] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:29.550098 22665 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:29.551895 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:29.552431 22742 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:29.555406 22665 raft_consensus.cc:2454] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 2.
I20240502 14:05:29.556310 22784 leader_election.cc:302] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:29.556921 22955 raft_consensus.cc:2781] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:29.557384 22955 raft_consensus.cc:686] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:29.557909 22742 raft_consensus.cc:2454] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 2.
I20240502 14:05:29.558133 22955 consensus_queue.cc:237] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:29.564352 22564 catalog_manager.cc:5484] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 1 to 2, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 2 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:29.949337 22665 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:29.950340 22958 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:29.962785 22742 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:29.963799 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:29.964346 22958 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:29.972199 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:29.972782 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:30.020211 22742 raft_consensus.cc:1261] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:30.021251 22958 consensus_queue.cc:1035] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:30.030045 22665 raft_consensus.cc:1261] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:30.031756 22973 consensus_queue.cc:1035] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:30.973688 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:30.979378 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:30.979714 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:32.012359 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:32.020700 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:32.021256 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:33.022080 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:33.027062 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:33.027391 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:05:33.040100 22597 debug-util.cc:398] Leaking SignalData structure 0x7b0800056d40 after lost signal to thread 22023
W20240502 14:05:33.040730 22597 debug-util.cc:398] Leaking SignalData structure 0x7b0800052420 after lost signal to thread 22600
W20240502 14:05:33.041406 22597 debug-util.cc:398] Leaking SignalData structure 0x7b08000bba20 after lost signal to thread 22693
W20240502 14:05:33.042045 22597 debug-util.cc:398] Leaking SignalData structure 0x7b0800156f60 after lost signal to thread 22768
W20240502 14:05:33.042659 22597 debug-util.cc:398] Leaking SignalData structure 0x7b08001fe0a0 after lost signal to thread 22843
W20240502 14:05:34.815429 22634 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.000s sys 0.007s
W20240502 14:05:34.816138 22634 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.000s sys 0.007s
I20240502 14:05:34.857048 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:34.880098 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:34.880684 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:34.898348 22984 raft_consensus.cc:482] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 653f39b212b2490ebb533ea039543f80)
I20240502 14:05:34.899180 22984 raft_consensus.cc:504] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:34.902125 22984 leader_election.cc:288] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719), 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409)
I20240502 14:05:34.905902 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "653f39b212b2490ebb533ea039543f80" is_pre_election: true
I20240502 14:05:34.921739 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec11102d790b4644a68db70d3e31e29d" is_pre_election: true
I20240502 14:05:34.929271 22710 leader_election.cc:302] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c95003c717f643558d1e9d072a91400e; no voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:34.930267 22989 raft_consensus.cc:2726] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:05:35.881455 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:35.887064 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:35.887434 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:05:35.909962 22840 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5e00 after lost signal to thread 22023
W20240502 14:05:35.910953 22840 debug-util.cc:398] Leaking SignalData structure 0x7b08001503a0 after lost signal to thread 22600
W20240502 14:05:35.912032 22840 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad4c0 after lost signal to thread 22693
W20240502 14:05:35.912968 22840 debug-util.cc:398] Leaking SignalData structure 0x7b0800141900 after lost signal to thread 22768
W20240502 14:05:35.913890 22840 debug-util.cc:398] Leaking SignalData structure 0x7b08000bbca0 after lost signal to thread 22843
I20240502 14:05:36.888221 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:36.894604 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:36.895013 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:37.895771 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:37.900837 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:37.901194 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:38.901894 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:38.906994 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:38.907361 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:39.930999 22020 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240502 14:05:39.943116 22818 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
mode: GRACEFUL
new_leader_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:52116
I20240502 14:05:39.943843 22818 raft_consensus.cc:595] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 LEADER]: Received request to transfer leadership to TS 653f39b212b2490ebb533ea039543f80
I20240502 14:05:39.948632 22818 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
mode: GRACEFUL
new_leader_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:52116
I20240502 14:05:39.949132 22818 raft_consensus.cc:595] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Received request to transfer leadership to TS 653f39b212b2490ebb533ea039543f80
I20240502 14:05:39.969627 22744 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
mode: GRACEFUL
new_leader_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:54986
I20240502 14:05:39.970054 22744 raft_consensus.cc:595] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Received request to transfer leadership to TS 653f39b212b2490ebb533ea039543f80
I20240502 14:05:39.973986 22744 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
mode: GRACEFUL
new_leader_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:54986
I20240502 14:05:39.974380 22744 raft_consensus.cc:595] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Received request to transfer leadership to TS 653f39b212b2490ebb533ea039543f80
I20240502 14:05:39.978061 22744 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
mode: GRACEFUL
new_leader_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:54986
I20240502 14:05:39.978453 22744 raft_consensus.cc:595] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Received request to transfer leadership to TS 653f39b212b2490ebb533ea039543f80
I20240502 14:05:40.036363 22996 raft_consensus.cc:979] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: : Instructing follower 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.036876 22987 raft_consensus.cc:1067] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Signalling peer 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.038223 22669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:40604
I20240502 14:05:40.038669 22669 raft_consensus.cc:482] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:40.038957 22669 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.044835 22669 raft_consensus.cc:504] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.046339 22669 leader_election.cc:288] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:40.047041 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:40.047341 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c7cfc908442e7a13a2a48ec4a6798" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:40.047612 22744 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.047880 22818 raft_consensus.cc:3032] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:40.048146 22818 raft_consensus.cc:729] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:40.048839 22818 consensus_queue.cc:260] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.049804 22818 raft_consensus.cc:3037] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.052661 22744 raft_consensus.cc:2454] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.053620 22633 leader_election.cc:302] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:40.054359 22991 raft_consensus.cc:2781] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:40.054852 22818 raft_consensus.cc:2454] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.055281 22991 raft_consensus.cc:686] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:40.056082 22991 consensus_queue.cc:237] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.064891 22563 catalog_manager.cc:5484] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 2 to 3, leader changed from ec11102d790b4644a68db70d3e31e29d (127.21.129.3) to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 3 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.111008 22977 raft_consensus.cc:979] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: : Instructing follower 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.111495 23003 raft_consensus.cc:1067] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Signalling peer 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.112674 22669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:40612
I20240502 14:05:40.113113 22669 raft_consensus.cc:482] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:40.113395 22669 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.118119 22669 raft_consensus.cc:504] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.119976 22669 leader_election.cc:288] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:40.120672 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:40.120811 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08463f4f86a945ed8169a8a468f0fb9a" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:40.121109 22744 raft_consensus.cc:3032] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:40.121441 22744 raft_consensus.cc:729] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:40.121554 22818 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.122040 22744 consensus_queue.cc:260] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.122997 22744 raft_consensus.cc:3037] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.127285 22818 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.128306 22635 leader_election.cc:302] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:40.128995 22991 raft_consensus.cc:2781] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:40.128996 22744 raft_consensus.cc:2454] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.129611 22991 raft_consensus.cc:686] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:40.130402 22991 consensus_queue.cc:237] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.136603 22563 catalog_manager.cc:5484] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 2 to 3, leader changed from c95003c717f643558d1e9d072a91400e (127.21.129.2) to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 3 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.280419 22977 raft_consensus.cc:979] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: : Instructing follower 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.280822 23003 raft_consensus.cc:1067] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Signalling peer 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.281980 22669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:40612
I20240502 14:05:40.282423 22669 raft_consensus.cc:482] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:40.282685 22669 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.287075 22669 raft_consensus.cc:504] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.288859 22669 leader_election.cc:288] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:40.289582 22987 raft_consensus.cc:979] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: : Instructing follower 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.289611 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:40.289700 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:40.290138 22996 raft_consensus.cc:1067] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 LEADER]: Signalling peer 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.290444 22744 raft_consensus.cc:3032] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:40.290504 22818 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.290884 22744 raft_consensus.cc:729] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:40.291580 22669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:40604
I20240502 14:05:40.292025 22669 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:40.291642 22744 consensus_queue.cc:260] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.292318 22669 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:40.292940 22744 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.296620 22669 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.296747 22818 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.297778 22635 leader_election.cc:302] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:40.298202 22669 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 4 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:40.298805 22991 raft_consensus.cc:2781] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:40.299332 22991 raft_consensus.cc:686] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:40.299285 22744 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.298918 22740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:40.300374 22740 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:40.300280 22991 consensus_queue.cc:237] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.300829 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:40.301343 22818 raft_consensus.cc:3032] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:05:40.301647 22818 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:40.302382 22818 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.303624 22818 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:40.305301 22740 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 4.
I20240502 14:05:40.306174 22633 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, c95003c717f643558d1e9d072a91400e; no voters:
I20240502 14:05:40.306888 22997 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:05:40.307349 22997 raft_consensus.cc:686] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:40.308625 22818 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 4.
I20240502 14:05:40.308161 22997 consensus_queue.cc:237] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.308408 22563 catalog_manager.cc:5484] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 2 to 3, leader changed from c95003c717f643558d1e9d072a91400e (127.21.129.2) to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 3 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.314342 22977 raft_consensus.cc:979] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: : Instructing follower 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.314745 23003 raft_consensus.cc:1067] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Signalling peer 653f39b212b2490ebb533ea039543f80 to start an election
I20240502 14:05:40.316216 22669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
from {username='slave'} at 127.0.0.1:40612
I20240502 14:05:40.316744 22669 raft_consensus.cc:482] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:40.317077 22669 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.321336 22563 catalog_manager.cc:5484] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 3 to 4, leader changed from ec11102d790b4644a68db70d3e31e29d (127.21.129.3) to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 4 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.321956 22669 raft_consensus.cc:504] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.323500 22669 leader_election.cc:288] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:40.323968 22740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:40.324592 22740 raft_consensus.cc:3032] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:05:40.324903 22740 raft_consensus.cc:729] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:40.325698 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "653f39b212b2490ebb533ea039543f80" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:40.325601 22740 consensus_queue.cc:260] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.326368 22818 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.327069 22740 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:05:40.332757 22818 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.333256 22740 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653f39b212b2490ebb533ea039543f80 in term 3.
I20240502 14:05:40.333837 22635 leader_election.cc:302] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653f39b212b2490ebb533ea039543f80, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:40.334502 22997 raft_consensus.cc:2781] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:05:40.334861 22997 raft_consensus.cc:686] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Becoming Leader. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:40.335597 22997 consensus_queue.cc:237] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:40.341234 22563 catalog_manager.cc:5484] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 reported cstate change: term changed from 2 to 3, leader changed from c95003c717f643558d1e9d072a91400e (127.21.129.2) to 653f39b212b2490ebb533ea039543f80 (127.21.129.1). New cstate: current_term: 3 leader_uuid: "653f39b212b2490ebb533ea039543f80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.524012 22818 raft_consensus.cc:1261] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.525033 22997 consensus_queue.cc:1035] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.536213 22740 raft_consensus.cc:1261] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.537815 23010 consensus_queue.cc:1035] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:05:40.579030 22818 raft_consensus.cc:1261] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.580196 23010 consensus_queue.cc:1035] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.590265 22740 raft_consensus.cc:1261] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.591624 22991 consensus_queue.cc:1035] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.712353 22818 raft_consensus.cc:1261] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.713490 22991 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.722247 22740 raft_consensus.cc:1261] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.723754 22991 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.834563 22740 raft_consensus.cc:1261] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.835561 22991 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.843914 22818 raft_consensus.cc:1261] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 14:05:40.845470 23017 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:05:40.885768 22818 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:40.886729 23017 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:40.894315 22740 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Refusing update from remote peer 653f39b212b2490ebb533ea039543f80: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:40.895864 22997 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:43.006196 22020 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240502 14:05:43.006732 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:43.013034 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:40624
I20240502 14:05:43.013500 22669 raft_consensus.cc:595] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Received request to transfer leadership to TS ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:43.015522 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:40624
I20240502 14:05:43.016002 22669 raft_consensus.cc:595] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 LEADER]: Received request to transfer leadership to TS c95003c717f643558d1e9d072a91400e
I20240502 14:05:43.018069 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:40624
I20240502 14:05:43.018239 23017 raft_consensus.cc:979] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: : Instructing follower ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:43.018581 22669 raft_consensus.cc:595] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Received request to transfer leadership to TS c95003c717f643558d1e9d072a91400e
I20240502 14:05:43.018721 23024 raft_consensus.cc:1067] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Signalling peer ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:43.020038 22818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:52094
I20240502 14:05:43.020476 22818 raft_consensus.cc:482] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:43.020570 22669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c"
dest_uuid: "653f39b212b2490ebb533ea039543f80"
mode: GRACEFUL
new_leader_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:40624
I20240502 14:05:43.020753 22818 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:43.021078 22669 raft_consensus.cc:595] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Received request to transfer leadership to TS ec11102d790b4644a68db70d3e31e29d
I20240502 14:05:43.021948 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 14:05:43.022354 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:43.025028 22818 raft_consensus.cc:504] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.026576 22818 leader_election.cc:288] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 4 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), c95003c717f643558d1e9d072a91400e (127.21.129.2:38639)
I20240502 14:05:43.027186 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:43.027540 22740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9d3139e124795aef9a4a59525495c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:43.027868 22669 raft_consensus.cc:3032] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:05:43.028044 22740 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:43.028231 22669 raft_consensus.cc:729] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:43.029006 22669 consensus_queue.cc:260] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.029923 22669 raft_consensus.cc:3037] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:43.030597 23017 raft_consensus.cc:979] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: : Instructing follower c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:43.031030 23024 raft_consensus.cc:1067] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 LEADER]: Signalling peer c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:43.032533 22744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:45410
I20240502 14:05:43.033056 22744 raft_consensus.cc:482] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:43.033396 22744 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:05:43.033561 22740 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 4.
I20240502 14:05:43.034705 22782 leader_election.cc:302] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:43.034952 22669 raft_consensus.cc:2454] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 4.
I20240502 14:05:43.035499 22996 raft_consensus.cc:2781] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:05:43.036531 22996 raft_consensus.cc:686] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 4 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:43.037261 22996 consensus_queue.cc:237] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.038831 22744 raft_consensus.cc:504] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.041249 22744 leader_election.cc:288] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 5 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:43.042038 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:43.042541 22818 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:05:43.042697 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f691ef6e0d546af865dbf2a186e5745" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:43.043262 22668 raft_consensus.cc:3032] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:05:43.043584 22668 raft_consensus.cc:729] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:43.044286 22668 consensus_queue.cc:260] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.044569 22564 catalog_manager.cc:5484] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 3 to 4, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 4 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:43.045504 22668 raft_consensus.cc:3037] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:05:43.046365 22818 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 5.
I20240502 14:05:43.047156 22710 leader_election.cc:302] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:43.047947 23003 raft_consensus.cc:2781] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:05:43.049181 23003 raft_consensus.cc:686] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 5 LEADER]: Becoming Leader. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:43.049999 23003 consensus_queue.cc:237] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.050747 22668 raft_consensus.cc:2454] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 5.
I20240502 14:05:43.058169 22563 catalog_manager.cc:5484] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e reported cstate change: term changed from 4 to 5, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to c95003c717f643558d1e9d072a91400e (127.21.129.2). New cstate: current_term: 5 leader_uuid: "c95003c717f643558d1e9d072a91400e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:43.171679 23017 raft_consensus.cc:979] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: : Instructing follower ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:43.172056 23024 raft_consensus.cc:1067] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Signalling peer ec11102d790b4644a68db70d3e31e29d to start an election
I20240502 14:05:43.173213 22818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c"
dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
from {username='slave'} at 127.0.0.1:52094
I20240502 14:05:43.173637 22818 raft_consensus.cc:482] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:43.173885 22818 raft_consensus.cc:3037] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:43.177896 22818 raft_consensus.cc:504] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:43.179270 22818 leader_election.cc:288] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 2 election: Requested vote from peers c95003c717f643558d1e9d072a91400e (127.21.129.2:38639), 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409)
I20240502 14:05:43.179913 22744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c95003c717f643558d1e9d072a91400e"
I20240502 14:05:43.180084 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0700467d454fadac8e1aea250bda7c" candidate_uuid: "ec11102d790b4644a68db70d3e31e29d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:43.180431 22744 raft_consensus.cc:3037] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:43.180667 22668 raft_consensus.cc:3032] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:05:43.180974 22668 raft_consensus.cc:729] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:43.181694 22668 consensus_queue.cc:260] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:43.182816 22668 raft_consensus.cc:3037] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:05:43.185387 22744 raft_consensus.cc:2454] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 2.
I20240502 14:05:43.186246 22782 leader_election.cc:302] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:43.187024 22996 raft_consensus.cc:2781] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:05:43.187500 22996 raft_consensus.cc:686] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Becoming Leader. State: Replica: ec11102d790b4644a68db70d3e31e29d, State: Running, Role: LEADER
I20240502 14:05:43.188254 22996 consensus_queue.cc:237] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } }
I20240502 14:05:43.188810 22668 raft_consensus.cc:2454] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec11102d790b4644a68db70d3e31e29d in term 2.
I20240502 14:05:43.214118 22563 catalog_manager.cc:5484] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d reported cstate change: term changed from 1 to 2, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to ec11102d790b4644a68db70d3e31e29d (127.21.129.3). New cstate: current_term: 2 leader_uuid: "ec11102d790b4644a68db70d3e31e29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:43.434963 22818 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 5 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 14:05:43.436055 23003 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:05:43.446640 22668 raft_consensus.cc:1261] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 5 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 14:05:43.453248 23003 consensus_queue.cc:1035] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:05:43.591231 23017 raft_consensus.cc:979] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: : Instructing follower c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:43.591768 23024 raft_consensus.cc:1067] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Signalling peer c95003c717f643558d1e9d072a91400e to start an election
I20240502 14:05:43.593143 22744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c"
dest_uuid: "c95003c717f643558d1e9d072a91400e"
from {username='slave'} at 127.0.0.1:45410
I20240502 14:05:43.593622 22744 raft_consensus.cc:482] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:05:43.593921 22744 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:43.600771 22744 raft_consensus.cc:504] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.602442 22744 leader_election.cc:288] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 4 election: Requested vote from peers 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409), ec11102d790b4644a68db70d3e31e29d (127.21.129.3:34719)
I20240502 14:05:43.603462 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "653f39b212b2490ebb533ea039543f80"
I20240502 14:05:43.603426 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae858579e7384cb392f6d655d83c6d2c" candidate_uuid: "c95003c717f643558d1e9d072a91400e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ec11102d790b4644a68db70d3e31e29d"
I20240502 14:05:43.604137 22668 raft_consensus.cc:3032] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:05:43.604246 22818 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:43.604440 22668 raft_consensus.cc:729] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 653f39b212b2490ebb533ea039543f80, State: Running, Role: LEADER
I20240502 14:05:43.606092 22668 consensus_queue.cc:260] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.607219 22669 raft_consensus.cc:1261] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:43.607565 22668 raft_consensus.cc:3037] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:05:43.608657 22996 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:43.610693 22818 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 4.
I20240502 14:05:43.611697 22710 leader_election.cc:302] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95003c717f643558d1e9d072a91400e, ec11102d790b4644a68db70d3e31e29d; no voters:
I20240502 14:05:43.612422 23003 raft_consensus.cc:2781] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:05:43.612843 23003 raft_consensus.cc:686] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 4 LEADER]: Becoming Leader. State: Replica: c95003c717f643558d1e9d072a91400e, State: Running, Role: LEADER
I20240502 14:05:43.613741 23003 consensus_queue.cc:237] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } }
I20240502 14:05:43.614969 22668 raft_consensus.cc:2454] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95003c717f643558d1e9d072a91400e in term 4.
I20240502 14:05:43.621789 22744 raft_consensus.cc:1261] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:43.627936 23025 consensus_queue.cc:1035] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:43.635059 22563 catalog_manager.cc:5484] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e reported cstate change: term changed from 3 to 4, leader changed from 653f39b212b2490ebb533ea039543f80 (127.21.129.1) to c95003c717f643558d1e9d072a91400e (127.21.129.2). New cstate: current_term: 4 leader_uuid: "c95003c717f643558d1e9d072a91400e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:43.658159 22744 raft_consensus.cc:1261] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:43.659184 22996 consensus_queue.cc:1035] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95003c717f643558d1e9d072a91400e" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:43.667465 22669 raft_consensus.cc:1261] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Refusing update from remote peer ec11102d790b4644a68db70d3e31e29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:05:43.668776 23043 consensus_queue.cc:1035] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:05:44.021147 22669 raft_consensus.cc:1261] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:44.022181 23048 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "653f39b212b2490ebb533ea039543f80" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:05:44.023227 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:44.030175 22817 raft_consensus.cc:1261] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Refusing update from remote peer c95003c717f643558d1e9d072a91400e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 14:05:44.031078 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:44.031688 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:44.031558 23028 consensus_queue.cc:1035] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11102d790b4644a68db70d3e31e29d" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:05:45.032603 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:45.037755 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:45.038152 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:46.038815 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:46.044075 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:46.044481 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:47.045106 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:47.051610 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:47.051990 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:48.052793 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:48.058032 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:48.058398 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:49.059090 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:49.066000 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:49.066373 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:50.067059 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:50.072046 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:50.072412 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:51.073088 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:51.078277 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:51.078637 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:52.079311 22020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:52.084255 22020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:52.084609 22020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:53.105656 22020 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240502 14:05:53.106158 22020 tablet_server.cc:178] TabletServer@127.21.129.1:0 shutting down...
W20240502 14:05:53.123366 22784 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 10 similar messages]
W20240502 14:05:53.128059 22784 consensus_peers.cc:485] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d -> Peer 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409): Couldn't send request to peer 653f39b212b2490ebb533ea039543f80. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:53.131595 22020 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:53.132203 22020 tablet_replica.cc:330] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.132761 22020 raft_consensus.cc:2229] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.133170 22020 raft_consensus.cc:2258] T 6f691ef6e0d546af865dbf2a186e5745 P 653f39b212b2490ebb533ea039543f80 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.134799 22020 tablet_replica.cc:330] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.135347 22020 raft_consensus.cc:2229] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.136050 22020 raft_consensus.cc:2258] T 08463f4f86a945ed8169a8a468f0fb9a P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.137590 22020 tablet_replica.cc:330] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.138082 22020 raft_consensus.cc:2229] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.138748 22020 raft_consensus.cc:2258] T 89cd480bdc5d47c985e46046cbde365a P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.140319 22020 tablet_replica.cc:330] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.140784 22020 raft_consensus.cc:2229] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.141176 22020 raft_consensus.cc:2258] T 5cfc9ac35956473bb83fbee4a2fe215e P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.142604 22020 tablet_replica.cc:330] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.143036 22020 raft_consensus.cc:2229] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.143445 22020 raft_consensus.cc:2258] T ae858579e7384cb392f6d655d83c6d2c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.144871 22020 tablet_replica.cc:330] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.145283 22020 raft_consensus.cc:2229] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.145918 22020 raft_consensus.cc:2258] T 502c7cfc908442e7a13a2a48ec4a6798 P 653f39b212b2490ebb533ea039543f80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.147370 22020 tablet_replica.cc:330] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.147796 22020 raft_consensus.cc:2229] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.148156 22020 raft_consensus.cc:2258] T 5ab9d3139e124795aef9a4a59525495c P 653f39b212b2490ebb533ea039543f80 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.149508 22020 tablet_replica.cc:330] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.149926 22020 raft_consensus.cc:2229] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.150314 22020 raft_consensus.cc:2258] T 03735110272046a8a958501c01b845ad P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.151758 22020 tablet_replica.cc:330] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.152138 22020 raft_consensus.cc:2229] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.152779 22020 raft_consensus.cc:2258] T ed6e932003b640b0bc213d24330c7223 P 653f39b212b2490ebb533ea039543f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.154198 22020 tablet_replica.cc:330] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80: stopping tablet replica
I20240502 14:05:53.154560 22020 raft_consensus.cc:2229] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.154875 22020 raft_consensus.cc:2258] T ee0700467d454fadac8e1aea250bda7c P 653f39b212b2490ebb533ea039543f80 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 14:05:53.160650 22710 consensus_peers.cc:485] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e -> Peer 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409): Couldn't send request to peer 653f39b212b2490ebb533ea039543f80. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:53.173012 22710 consensus_peers.cc:485] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e -> Peer 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409): Couldn't send request to peer 653f39b212b2490ebb533ea039543f80. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.1:46409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:53.183779 22020 tablet_server.cc:194] TabletServer@127.21.129.1:0 shutdown complete.
I20240502 14:05:53.199838 22020 tablet_server.cc:178] TabletServer@127.21.129.2:0 shutting down...
W20240502 14:05:53.205834 22782 consensus_peers.cc:485] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d -> Peer c95003c717f643558d1e9d072a91400e (127.21.129.2:38639): Couldn't send request to peer c95003c717f643558d1e9d072a91400e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:53.236202 22020 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:53.237212 22020 tablet_replica.cc:330] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.238476 22020 raft_consensus.cc:2229] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.239866 22020 raft_consensus.cc:2258] T 5cfc9ac35956473bb83fbee4a2fe215e P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.242146 22020 tablet_replica.cc:330] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.242702 22020 raft_consensus.cc:2229] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 4 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.243968 22020 raft_consensus.cc:2258] T ae858579e7384cb392f6d655d83c6d2c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.245834 22020 tablet_replica.cc:330] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.246277 22020 raft_consensus.cc:2229] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 5 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.247370 22020 raft_consensus.cc:2258] T 6f691ef6e0d546af865dbf2a186e5745 P c95003c717f643558d1e9d072a91400e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.249214 22020 tablet_replica.cc:330] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.249871 22020 raft_consensus.cc:2229] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.250573 22020 raft_consensus.cc:2258] T ee0700467d454fadac8e1aea250bda7c P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.252532 22020 tablet_replica.cc:330] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.253392 22020 raft_consensus.cc:2229] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.254179 22020 raft_consensus.cc:2258] T 08463f4f86a945ed8169a8a468f0fb9a P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.256989 22020 tablet_replica.cc:330] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.257578 22020 raft_consensus.cc:2229] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.258186 22020 raft_consensus.cc:2258] T 89cd480bdc5d47c985e46046cbde365a P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.260298 22020 tablet_replica.cc:330] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.260792 22020 raft_consensus.cc:2229] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.261262 22020 raft_consensus.cc:2258] T 03735110272046a8a958501c01b845ad P c95003c717f643558d1e9d072a91400e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.262856 22020 tablet_replica.cc:330] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.263396 22020 raft_consensus.cc:2229] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.263831 22020 raft_consensus.cc:2258] T ed6e932003b640b0bc213d24330c7223 P c95003c717f643558d1e9d072a91400e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.265573 22020 tablet_replica.cc:330] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.266023 22020 raft_consensus.cc:2229] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.266543 22020 raft_consensus.cc:2258] T 5ab9d3139e124795aef9a4a59525495c P c95003c717f643558d1e9d072a91400e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.268453 22020 tablet_replica.cc:330] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e: stopping tablet replica
I20240502 14:05:53.268860 22020 raft_consensus.cc:2229] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.269344 22020 raft_consensus.cc:2258] T 502c7cfc908442e7a13a2a48ec4a6798 P c95003c717f643558d1e9d072a91400e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.299057 22020 tablet_server.cc:194] TabletServer@127.21.129.2:0 shutdown complete.
I20240502 14:05:53.318054 22020 tablet_server.cc:178] TabletServer@127.21.129.3:0 shutting down...
I20240502 14:05:53.341773 22020 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:53.342357 22020 tablet_replica.cc:330] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.343542 22020 raft_consensus.cc:2229] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.344241 22020 raft_consensus.cc:2258] T 502c7cfc908442e7a13a2a48ec4a6798 P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.346807 22020 tablet_replica.cc:330] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.347875 22020 raft_consensus.cc:2229] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.348861 22020 raft_consensus.cc:2258] T 6f691ef6e0d546af865dbf2a186e5745 P ec11102d790b4644a68db70d3e31e29d [term 5 FOLLOWER]: Raft consensus is shut down!
W20240502 14:05:53.350265 22784 consensus_peers.cc:485] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d -> Peer 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409): Couldn't send request to peer 653f39b212b2490ebb533ea039543f80. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.1:46409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:53.351502 22020 tablet_replica.cc:330] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.352223 22020 raft_consensus.cc:2229] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 4 LEADER]: Raft consensus shutting down.
W20240502 14:05:53.352640 22784 consensus_peers.cc:485] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d -> Peer 653f39b212b2490ebb533ea039543f80 (127.21.129.1:46409): Couldn't send request to peer 653f39b212b2490ebb533ea039543f80. Status: Network error: Client connection negotiation failed: client connection to 127.21.129.1:46409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:53.353610 22020 raft_consensus.cc:2258] T 5ab9d3139e124795aef9a4a59525495c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.355365 22020 tablet_replica.cc:330] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.355859 22020 raft_consensus.cc:2229] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.356256 22020 raft_consensus.cc:2258] T ed6e932003b640b0bc213d24330c7223 P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.357846 22020 tablet_replica.cc:330] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.358309 22020 raft_consensus.cc:2229] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.359249 22020 raft_consensus.cc:2258] T 03735110272046a8a958501c01b845ad P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.360870 22020 tablet_replica.cc:330] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.361296 22020 raft_consensus.cc:2229] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.361759 22020 raft_consensus.cc:2258] T 08463f4f86a945ed8169a8a468f0fb9a P ec11102d790b4644a68db70d3e31e29d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.363432 22020 tablet_replica.cc:330] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.363839 22020 raft_consensus.cc:2229] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.364295 22020 raft_consensus.cc:2258] T 89cd480bdc5d47c985e46046cbde365a P ec11102d790b4644a68db70d3e31e29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.365844 22020 tablet_replica.cc:330] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.366256 22020 raft_consensus.cc:2229] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.366735 22020 raft_consensus.cc:2258] T ae858579e7384cb392f6d655d83c6d2c P ec11102d790b4644a68db70d3e31e29d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.368274 22020 tablet_replica.cc:330] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.368677 22020 raft_consensus.cc:2229] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:53.369097 22020 raft_consensus.cc:2258] T 5cfc9ac35956473bb83fbee4a2fe215e P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.370615 22020 tablet_replica.cc:330] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d: stopping tablet replica
I20240502 14:05:53.371013 22020 raft_consensus.cc:2229] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.371857 22020 raft_consensus.cc:2258] T ee0700467d454fadac8e1aea250bda7c P ec11102d790b4644a68db70d3e31e29d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.416425 22020 tablet_server.cc:194] TabletServer@127.21.129.3:0 shutdown complete.
I20240502 14:05:53.434032 22020 master.cc:537] Master@127.21.129.62:44415 shutting down...
I20240502 14:05:53.453817 22020 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:53.454349 22020 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:53.454625 22020 tablet_replica.cc:330] T 00000000000000000000000000000000 P 29f24112f9fd41b5b792e51518bc64cf: stopping tablet replica
I20240502 14:05:53.486491 22020 master.cc:552] Master@127.21.129.62:44415 shutdown complete.
653f39b212b2490ebb533ea039543f80 5
c95003c717f643558d1e9d072a91400e 4
ec11102d790b4644a68db70d3e31e29d 1
653f39b212b2490ebb533ea039543f80 3
c95003c717f643558d1e9d072a91400e 4
ec11102d790b4644a68db70d3e31e29d 3
653f39b212b2490ebb533ea039543f80 8
c95003c717f643558d1e9d072a91400e 1
ec11102d790b4644a68db70d3e31e29d 1
653f39b212b2490ebb533ea039543f80 4
c95003c717f643558d1e9d072a91400e 3
ec11102d790b4644a68db70d3e31e29d 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29285 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240502 14:05:53.526880 22020 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.129.62:37293
I20240502 14:05:53.527946 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:53.533947 23068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.534052 23067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.535104 23070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:53.535739 22020 server_base.cc:1034] running on GCE node
I20240502 14:05:53.536589 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:53.536777 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:53.536912 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658753536895 us; error 0 us; skew 500 ppm
I20240502 14:05:53.537395 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:53.544817 22020 webserver.cc:460] Webserver started at http://127.21.129.62:35435/ using document root <none> and password file <none>
I20240502 14:05:53.545250 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:53.545418 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:53.545673 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:53.546854 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/master-0-root/instance:
uuid: "6438954d5fdf441590ba4eb51cdd080f"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.551959 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:05:53.556077 23075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.556951 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:05:53.557281 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/master-0-root
uuid: "6438954d5fdf441590ba4eb51cdd080f"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.557626 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:53.589999 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:53.591421 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:53.628218 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.62:37293
I20240502 14:05:53.628314 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.62:37293 every 8 connection(s)
I20240502 14:05:53.631839 23129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:53.641042 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f: Bootstrap starting.
I20240502 14:05:53.645134 23129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:53.649749 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f: No bootstrap required, opened a new log
I20240502 14:05:53.651695 23129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6438954d5fdf441590ba4eb51cdd080f" member_type: VOTER }
I20240502 14:05:53.652148 23129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:53.652360 23129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6438954d5fdf441590ba4eb51cdd080f, State: Initialized, Role: FOLLOWER
I20240502 14:05:53.652894 23129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6438954d5fdf441590ba4eb51cdd080f" member_type: VOTER }
I20240502 14:05:53.653348 23129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:53.653561 23129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:53.653810 23129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:53.657306 23129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6438954d5fdf441590ba4eb51cdd080f" member_type: VOTER }
I20240502 14:05:53.657765 23129 leader_election.cc:302] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6438954d5fdf441590ba4eb51cdd080f; no voters:
I20240502 14:05:53.658802 23129 leader_election.cc:288] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:53.659161 23132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:53.660466 23132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [term 1 LEADER]: Becoming Leader. State: Replica: 6438954d5fdf441590ba4eb51cdd080f, State: Running, Role: LEADER
I20240502 14:05:53.661093 23132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6438954d5fdf441590ba4eb51cdd080f" member_type: VOTER }
I20240502 14:05:53.661643 23129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:53.667007 23134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6438954d5fdf441590ba4eb51cdd080f. Latest consensus state: current_term: 1 leader_uuid: "6438954d5fdf441590ba4eb51cdd080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6438954d5fdf441590ba4eb51cdd080f" member_type: VOTER } }
I20240502 14:05:53.667698 23134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:53.669413 23133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6438954d5fdf441590ba4eb51cdd080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6438954d5fdf441590ba4eb51cdd080f" member_type: VOTER } }
I20240502 14:05:53.669951 23133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:53.673481 23142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:53.677829 23142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:53.678778 22020 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:05:53.685426 23142 catalog_manager.cc:1347] Generated new cluster ID: 4109c74595754bec99ad1e767e2e8bfa
I20240502 14:05:53.685616 23142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:53.716768 23142 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:53.717808 23142 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:53.727301 23142 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6438954d5fdf441590ba4eb51cdd080f: Generated new TSK 0
I20240502 14:05:53.727752 23142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:53.744315 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:53.749459 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.750842 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.752013 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:53.752525 22020 server_base.cc:1034] running on GCE node
I20240502 14:05:53.753340 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:53.753520 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:53.753656 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658753753640 us; error 0 us; skew 500 ppm
I20240502 14:05:53.754173 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:53.756260 22020 webserver.cc:460] Webserver started at http://127.21.129.1:40241/ using document root <none> and password file <none>
I20240502 14:05:53.756691 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:53.756855 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:53.757097 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:53.758148 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root/instance:
uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.762324 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:05:53.765318 23158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.765978 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:05:53.766217 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root
uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.766480 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:53.787811 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:53.788836 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:53.790161 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:53.792258 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:53.792443 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.792665 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:53.792824 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.829891 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.1:38171
I20240502 14:05:53.830019 23221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.1:38171 every 8 connection(s)
I20240502 14:05:53.834564 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:53.840843 23226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.843926 23227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.845583 23229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:53.847440 23222 heartbeater.cc:347] Connected to a master server at 127.21.129.62:37293
I20240502 14:05:53.847815 23222 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:53.848302 22020 server_base.cc:1034] running on GCE node
I20240502 14:05:53.848697 23222 heartbeater.cc:511] Master 127.21.129.62:37293 requested a full tablet report, sending...
I20240502 14:05:53.849222 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:53.849431 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:53.849571 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658753849556 us; error 0 us; skew 500 ppm
I20240502 14:05:53.850131 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:53.850813 23092 ts_manager.cc:194] Registered new tserver with Master: cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:05:53.852577 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56060
I20240502 14:05:53.852727 22020 webserver.cc:460] Webserver started at http://127.21.129.2:39071/ using document root <none> and password file <none>
I20240502 14:05:53.853248 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:53.853422 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:53.853713 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:53.854719 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root/instance:
uuid: "82490d85cd3d4a1180dc3ecabdd88598"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.858846 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:05:53.861963 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.862741 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:53.863001 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root
uuid: "82490d85cd3d4a1180dc3ecabdd88598"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.863284 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:53.885532 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:53.886571 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:53.887894 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:53.889892 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:53.890079 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.890276 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:53.890429 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.926277 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.2:34169
I20240502 14:05:53.926380 23297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.2:34169 every 8 connection(s)
I20240502 14:05:53.930639 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:53.936803 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:53.939373 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:53.940642 22020 server_base.cc:1034] running on GCE node
W20240502 14:05:53.943069 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:53.943553 23298 heartbeater.cc:347] Connected to a master server at 127.21.129.62:37293
I20240502 14:05:53.943943 23298 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:53.944072 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:53.944352 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:53.944569 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658753944550 us; error 0 us; skew 500 ppm
I20240502 14:05:53.944751 23298 heartbeater.cc:511] Master 127.21.129.62:37293 requested a full tablet report, sending...
I20240502 14:05:53.945307 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:53.947072 23092 ts_manager.cc:194] Registered new tserver with Master: 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:53.948603 22020 webserver.cc:460] Webserver started at http://127.21.129.3:43913/ using document root <none> and password file <none>
I20240502 14:05:53.948691 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56074
I20240502 14:05:53.949082 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:53.949278 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:53.949523 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:53.950589 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root/instance:
uuid: "980e09c2959e408db93d72b7125ccf63"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.954751 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:05:53.957693 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.958400 22020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:53.958664 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root
uuid: "980e09c2959e408db93d72b7125ccf63"
format_stamp: "Formatted at 2024-05-02 14:05:53 on dist-test-slave-bdl7"
I20240502 14:05:53.958915 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:53.971089 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:53.972110 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:53.973414 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:53.975497 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:53.975687 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:53.975903 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:53.976053 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:54.012831 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.3:45179
I20240502 14:05:54.012970 23371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.3:45179 every 8 connection(s)
I20240502 14:05:54.024935 23372 heartbeater.cc:347] Connected to a master server at 127.21.129.62:37293
I20240502 14:05:54.025246 23372 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:54.025938 23372 heartbeater.cc:511] Master 127.21.129.62:37293 requested a full tablet report, sending...
I20240502 14:05:54.027594 23092 ts_manager.cc:194] Registered new tserver with Master: 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:54.027772 22020 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012034386s
I20240502 14:05:54.028856 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56082
I20240502 14:05:54.030051 22020 test_util.cc:268] Using random seed: 435284430
I20240502 14:05:54.050418 23092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56084:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:05:54.056385 23092 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:05:54.243382 23262 tablet_service.cc:1467] Processing CreateTablet for tablet 58b50f554edf4267b72624ec5b865c2e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 14:05:54.243671 23261 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee3ba4c1e8843d8b94029ece423580f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 14:05:54.244774 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b50f554edf4267b72624ec5b865c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.245744 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee3ba4c1e8843d8b94029ece423580f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.248324 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5069e564364a3b82e55d410fc03cec (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 14:05:54.249588 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5069e564364a3b82e55d410fc03cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.249646 23337 tablet_service.cc:1467] Processing CreateTablet for tablet 58b50f554edf4267b72624ec5b865c2e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 14:05:54.250878 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b50f554edf4267b72624ec5b865c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.254179 23259 tablet_service.cc:1467] Processing CreateTablet for tablet ff6e89be70bb4088ae2fb35610efbbbc (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 14:05:54.255416 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff6e89be70bb4088ae2fb35610efbbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.260660 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 0e158eeeafc149a58edb5538c52bbbdf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 14:05:54.261307 23336 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee3ba4c1e8843d8b94029ece423580f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 14:05:54.261878 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e158eeeafc149a58edb5538c52bbbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.262521 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee3ba4c1e8843d8b94029ece423580f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.267426 23257 tablet_service.cc:1467] Processing CreateTablet for tablet ac80905fa745417489dcee47a31635fa (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 14:05:54.268640 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac80905fa745417489dcee47a31635fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.274960 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 009254b7af044911807829cd2423373e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 14:05:54.276157 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009254b7af044911807829cd2423373e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.278378 23335 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5069e564364a3b82e55d410fc03cec (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 14:05:54.279599 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5069e564364a3b82e55d410fc03cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.280709 23255 tablet_service.cc:1467] Processing CreateTablet for tablet 35e4ee45e707479b874fab4903df88f6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 14:05:54.281908 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e4ee45e707479b874fab4903df88f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.284265 23332 tablet_service.cc:1467] Processing CreateTablet for tablet ac80905fa745417489dcee47a31635fa (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 14:05:54.285423 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac80905fa745417489dcee47a31635fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.288144 23328 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8d132cd4e545b29c79863b5aa25f5f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 14:05:54.289330 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8d132cd4e545b29c79863b5aa25f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.292253 23331 tablet_service.cc:1467] Processing CreateTablet for tablet 009254b7af044911807829cd2423373e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 14:05:54.293471 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009254b7af044911807829cd2423373e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.300351 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c31e65498481696374e87a81890fa (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 14:05:54.301532 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c31e65498481696374e87a81890fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.304262 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c31e65498481696374e87a81890fa (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 14:05:54.305442 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c31e65498481696374e87a81890fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.308279 23253 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8d132cd4e545b29c79863b5aa25f5f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 14:05:54.309480 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8d132cd4e545b29c79863b5aa25f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.312247 23330 tablet_service.cc:1467] Processing CreateTablet for tablet 35e4ee45e707479b874fab4903df88f6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 14:05:54.313421 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e4ee45e707479b874fab4903df88f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.316130 23334 tablet_service.cc:1467] Processing CreateTablet for tablet ff6e89be70bb4088ae2fb35610efbbbc (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 14:05:54.317322 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff6e89be70bb4088ae2fb35610efbbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.318725 23337 tablet_service.cc:1467] Processing CreateTablet for tablet e7774595b1544e5391939b585e1f40c8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 14:05:54.323959 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7774595b1544e5391939b585e1f40c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.283036 23333 tablet_service.cc:1467] Processing CreateTablet for tablet 0e158eeeafc149a58edb5538c52bbbdf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 14:05:54.339907 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e158eeeafc149a58edb5538c52bbbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.352032 23389 tablet_bootstrap.cc:492] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:54.358247 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 58b50f554edf4267b72624ec5b865c2e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 14:05:54.359460 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b50f554edf4267b72624ec5b865c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.364332 23185 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee3ba4c1e8843d8b94029ece423580f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 14:05:54.364568 23336 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9e5363c783444bbb4f631a0ee208d9 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 14:05:54.364701 23183 tablet_service.cc:1467] Processing CreateTablet for tablet ff6e89be70bb4088ae2fb35610efbbbc (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 14:05:54.365571 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee3ba4c1e8843d8b94029ece423580f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.365655 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9e5363c783444bbb4f631a0ee208d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.366895 23181 tablet_service.cc:1467] Processing CreateTablet for tablet ac80905fa745417489dcee47a31635fa (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 14:05:54.368252 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac80905fa745417489dcee47a31635fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.372411 23182 tablet_service.cc:1467] Processing CreateTablet for tablet 0e158eeeafc149a58edb5538c52bbbdf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 14:05:54.373716 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e158eeeafc149a58edb5538c52bbbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.376672 23389 tablet_bootstrap.cc:654] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:54.376363 23179 tablet_service.cc:1467] Processing CreateTablet for tablet 35e4ee45e707479b874fab4903df88f6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 14:05:54.377993 23178 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c31e65498481696374e87a81890fa (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 14:05:54.379966 23336 tablet_service.cc:1467] Processing CreateTablet for tablet 9c72f195f8ab449e939a764c5d237d81 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 14:05:54.383625 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff6e89be70bb4088ae2fb35610efbbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.408258 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e4ee45e707479b874fab4903df88f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.411491 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c31e65498481696374e87a81890fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.384714 23259 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9e5363c783444bbb4f631a0ee208d9 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 14:05:54.390636 23184 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5069e564364a3b82e55d410fc03cec (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 14:05:54.394608 23253 tablet_service.cc:1467] Processing CreateTablet for tablet 9c72f195f8ab449e939a764c5d237d81 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 14:05:54.399907 23255 tablet_service.cc:1467] Processing CreateTablet for tablet e7774595b1544e5391939b585e1f40c8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 14:05:54.402307 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 71b15d9239134710bfe520f70f80a2b6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 14:05:54.427793 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5069e564364a3b82e55d410fc03cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.402478 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 71b15d9239134710bfe520f70f80a2b6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 14:05:54.402958 23337 tablet_service.cc:1467] Processing CreateTablet for tablet eb3ff7f6ffd34055abb266d2b25e2e4d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 14:05:54.404100 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 009254b7af044911807829cd2423373e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 14:05:54.428308 23261 tablet_service.cc:1467] Processing CreateTablet for tablet eb3ff7f6ffd34055abb266d2b25e2e4d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 14:05:54.429164 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009254b7af044911807829cd2423373e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.404259 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8d132cd4e545b29c79863b5aa25f5f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 14:05:54.426937 23388 tablet_bootstrap.cc:492] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:54.432660 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c72f195f8ab449e939a764c5d237d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.432773 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8d132cd4e545b29c79863b5aa25f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.444515 23262 tablet_service.cc:1467] Processing CreateTablet for tablet cccf9e842a284cf8bcc8d60d0dd1a22b (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 14:05:54.445590 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ff7f6ffd34055abb266d2b25e2e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.446333 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9e5363c783444bbb4f631a0ee208d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.448726 23331 tablet_service.cc:1467] Processing CreateTablet for tablet cccf9e842a284cf8bcc8d60d0dd1a22b (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 14:05:54.451289 23334 tablet_service.cc:1467] Processing CreateTablet for tablet b1f8be05930c456fa1f16fe18fdc7d92 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 14:05:54.465742 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b15d9239134710bfe520f70f80a2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.466905 23333 tablet_service.cc:1467] Processing CreateTablet for tablet 271b7519a6f04979b29317f144bc70fe (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 14:05:54.460891 23388 tablet_bootstrap.cc:654] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:54.467736 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7774595b1544e5391939b585e1f40c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.468403 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f8be05930c456fa1f16fe18fdc7d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.469663 23328 tablet_service.cc:1467] Processing CreateTablet for tablet dc011e53d9184101a0e9b06c61f73c93 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 14:05:54.471416 23257 tablet_service.cc:1467] Processing CreateTablet for tablet b1f8be05930c456fa1f16fe18fdc7d92 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 14:05:54.453471 23332 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2ba63147fa4983ae30a46ed72e9568 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 14:05:54.473070 23330 tablet_service.cc:1467] Processing CreateTablet for tablet e5c7a15545d94e5c8931cb1c26d3f3db (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 14:05:54.478278 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf9e842a284cf8bcc8d60d0dd1a22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.474627 23335 tablet_service.cc:1467] Processing CreateTablet for tablet 12b5382d476949269a01d25f2e60c2d8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 14:05:54.481719 23254 tablet_service.cc:1467] Processing CreateTablet for tablet dc011e53d9184101a0e9b06c61f73c93 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 14:05:54.482934 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271b7519a6f04979b29317f144bc70fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.496335 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2ba63147fa4983ae30a46ed72e9568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.487478 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2ba63147fa4983ae30a46ed72e9568 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 14:05:54.499531 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 271b7519a6f04979b29317f144bc70fe (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 14:05:54.503000 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b5382d476949269a01d25f2e60c2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.491667 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c72f195f8ab449e939a764c5d237d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.503599 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc011e53d9184101a0e9b06c61f73c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.504149 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b15d9239134710bfe520f70f80a2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.506489 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ff7f6ffd34055abb266d2b25e2e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.507468 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f8be05930c456fa1f16fe18fdc7d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.492867 23179 tablet_service.cc:1467] Processing CreateTablet for tablet e7774595b1544e5391939b585e1f40c8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 14:05:54.511487 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c7a15545d94e5c8931cb1c26d3f3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.512008 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7774595b1544e5391939b585e1f40c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.512653 23389 tablet_bootstrap.cc:492] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:54.493101 23185 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9e5363c783444bbb4f631a0ee208d9 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 14:05:54.513054 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2ba63147fa4983ae30a46ed72e9568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.495142 23184 tablet_service.cc:1467] Processing CreateTablet for tablet 9c72f195f8ab449e939a764c5d237d81 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 14:05:54.514837 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9e5363c783444bbb4f631a0ee208d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.519805 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c72f195f8ab449e939a764c5d237d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.520393 23178 tablet_service.cc:1467] Processing CreateTablet for tablet 71b15d9239134710bfe520f70f80a2b6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 14:05:54.521538 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b15d9239134710bfe520f70f80a2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.513099 23389 ts_tablet_manager.cc:1367] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.007s
I20240502 14:05:54.529783 23177 tablet_service.cc:1467] Processing CreateTablet for tablet eb3ff7f6ffd34055abb266d2b25e2e4d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 14:05:54.530964 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ff7f6ffd34055abb266d2b25e2e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.532640 23334 tablet_service.cc:1467] Processing CreateTablet for tablet 2630762fac4d4b6c99da94bc7b0f77cf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 14:05:54.532931 23183 tablet_service.cc:1467] Processing CreateTablet for tablet b1f8be05930c456fa1f16fe18fdc7d92 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 14:05:54.533829 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2630762fac4d4b6c99da94bc7b0f77cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.530233 23180 tablet_service.cc:1467] Processing CreateTablet for tablet cccf9e842a284cf8bcc8d60d0dd1a22b (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 14:05:54.536005 23336 tablet_service.cc:1467] Processing CreateTablet for tablet a53a02221c4845399218ffb0554d4044 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 14:05:54.541915 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf9e842a284cf8bcc8d60d0dd1a22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.536718 23259 tablet_service.cc:1467] Processing CreateTablet for tablet e5c7a15545d94e5c8931cb1c26d3f3db (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 14:05:54.548128 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f8be05930c456fa1f16fe18fdc7d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.545298 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf9e842a284cf8bcc8d60d0dd1a22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.555635 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc011e53d9184101a0e9b06c61f73c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.556479 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c7a15545d94e5c8931cb1c26d3f3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.561472 23255 tablet_service.cc:1467] Processing CreateTablet for tablet 12b5382d476949269a01d25f2e60c2d8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 14:05:54.562898 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53a02221c4845399218ffb0554d4044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.570093 23389 raft_consensus.cc:348] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.595346 23389 raft_consensus.cc:374] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:54.595738 23389 raft_consensus.cc:729] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:54.583776 23391 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:54.584846 23182 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2ba63147fa4983ae30a46ed72e9568 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 14:05:54.597914 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2ba63147fa4983ae30a46ed72e9568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.597712 23389 consensus_queue.cc:260] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.582410 23331 tablet_service.cc:1467] Processing CreateTablet for tablet 77cb09543bbb470883b667e0303657b6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 14:05:54.599476 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271b7519a6f04979b29317f144bc70fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.592653 23181 tablet_service.cc:1467] Processing CreateTablet for tablet dc011e53d9184101a0e9b06c61f73c93 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 14:05:54.600409 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc011e53d9184101a0e9b06c61f73c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.602330 23391 tablet_bootstrap.cc:654] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:54.585702 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 271b7519a6f04979b29317f144bc70fe (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 14:05:54.580809 23328 tablet_service.cc:1467] Processing CreateTablet for tablet b400e3ddeee74798b411ee7a4e285b7a (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 14:05:54.604483 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271b7519a6f04979b29317f144bc70fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.604535 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cb09543bbb470883b667e0303657b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.607458 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b400e3ddeee74798b411ee7a4e285b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.608737 23333 tablet_service.cc:1467] Processing CreateTablet for tablet fb1848b778274b788853fcb99a639b88 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 14:05:54.609949 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1848b778274b788853fcb99a639b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.596577 23337 tablet_service.cc:1467] Processing CreateTablet for tablet eac6f78a24a0432cae83b3b832145059 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 14:05:54.604563 23332 tablet_service.cc:1467] Processing CreateTablet for tablet cc8b4f12c6d843efbcc7359b62a52cf5 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 14:05:54.611847 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eac6f78a24a0432cae83b3b832145059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.612664 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8b4f12c6d843efbcc7359b62a52cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.613014 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b5382d476949269a01d25f2e60c2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.625969 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 2630762fac4d4b6c99da94bc7b0f77cf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 14:05:54.627657 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 06cf069882cc4e469cd068f6da62ecbb (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 14:05:54.611573 23185 tablet_service.cc:1467] Processing CreateTablet for tablet e5c7a15545d94e5c8931cb1c26d3f3db (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 14:05:54.634699 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cf069882cc4e469cd068f6da62ecbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.644277 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c7a15545d94e5c8931cb1c26d3f3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.645094 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2630762fac4d4b6c99da94bc7b0f77cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.645684 23388 tablet_bootstrap.cc:492] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:54.651613 23388 ts_tablet_manager.cc:1367] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.225s user 0.014s sys 0.022s
I20240502 14:05:54.653262 23257 tablet_service.cc:1467] Processing CreateTablet for tablet b400e3ddeee74798b411ee7a4e285b7a (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 14:05:54.654397 23388 raft_consensus.cc:348] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:54.655184 23388 raft_consensus.cc:374] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:54.655722 23388 raft_consensus.cc:729] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:54.656502 23388 consensus_queue.cc:260] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:54.648473 23391 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:54.655004 23179 tablet_service.cc:1467] Processing CreateTablet for tablet 2630762fac4d4b6c99da94bc7b0f77cf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 14:05:54.649466 23256 tablet_service.cc:1467] Processing CreateTablet for tablet a53a02221c4845399218ffb0554d4044 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 14:05:54.658515 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2630762fac4d4b6c99da94bc7b0f77cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.649935 23330 tablet_service.cc:1467] Processing CreateTablet for tablet e6a156a5f03e4c8c97f11fd328ad5018 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 14:05:54.663471 23391 ts_tablet_manager.cc:1367] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.000s
I20240502 14:05:54.675503 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a156a5f03e4c8c97f11fd328ad5018. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.676793 23184 tablet_service.cc:1467] Processing CreateTablet for tablet a53a02221c4845399218ffb0554d4044 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 14:05:54.677987 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53a02221c4845399218ffb0554d4044. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:05:54.681447 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:05:54.683259 23183 tablet_service.cc:1467] Processing CreateTablet for tablet b400e3ddeee74798b411ee7a4e285b7a (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 14:05:54.684476 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b400e3ddeee74798b411ee7a4e285b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.674974 23262 tablet_service.cc:1467] Processing CreateTablet for tablet 77cb09543bbb470883b667e0303657b6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 14:05:54.689427 23178 tablet_service.cc:1467] Processing CreateTablet for tablet 77cb09543bbb470883b667e0303657b6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 14:05:54.690557 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cb09543bbb470883b667e0303657b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.692152 23335 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5b811e321d4679bb9a64f5965c6a7d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 14:05:54.695086 23180 tablet_service.cc:1467] Processing CreateTablet for tablet eac6f78a24a0432cae83b3b832145059 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 14:05:54.696259 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eac6f78a24a0432cae83b3b832145059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.696489 23261 tablet_service.cc:1467] Processing CreateTablet for tablet eac6f78a24a0432cae83b3b832145059 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 14:05:54.704234 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eac6f78a24a0432cae83b3b832145059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.704840 23372 heartbeater.cc:503] Master 127.21.129.62:37293 was elected leader, sending a full tablet report...
I20240502 14:05:54.705729 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b400e3ddeee74798b411ee7a4e285b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.708101 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53a02221c4845399218ffb0554d4044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.709262 23389 ts_tablet_manager.cc:1398] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.166s user 0.004s sys 0.004s
I20240502 14:05:54.710748 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cb09543bbb470883b667e0303657b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.719413 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 06cf069882cc4e469cd068f6da62ecbb (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 14:05:54.719806 23388 ts_tablet_manager.cc:1398] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.068s user 0.010s sys 0.008s
I20240502 14:05:54.715822 23185 tablet_service.cc:1467] Processing CreateTablet for tablet e6a156a5f03e4c8c97f11fd328ad5018 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 14:05:54.720382 23181 tablet_service.cc:1467] Processing CreateTablet for tablet 06cf069882cc4e469cd068f6da62ecbb (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 14:05:54.720886 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a156a5f03e4c8c97f11fd328ad5018. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.721498 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cf069882cc4e469cd068f6da62ecbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.731465 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5b811e321d4679bb9a64f5965c6a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.731825 23334 tablet_service.cc:1467] Processing CreateTablet for tablet b90f20f57cf34057a5e3301a044ad3ca (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 14:05:54.733054 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90f20f57cf34057a5e3301a044ad3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.729995 23298 heartbeater.cc:503] Master 127.21.129.62:37293 was elected leader, sending a full tablet report...
I20240502 14:05:54.735419 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 12b5382d476949269a01d25f2e60c2d8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 14:05:54.736570 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b5382d476949269a01d25f2e60c2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.737236 23391 raft_consensus.cc:348] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.737985 23391 raft_consensus.cc:374] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:54.738271 23391 raft_consensus.cc:729] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:54.739077 23391 consensus_queue.cc:260] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.747438 23259 tablet_service.cc:1467] Processing CreateTablet for tablet e6a156a5f03e4c8c97f11fd328ad5018 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 14:05:54.704778 23179 tablet_service.cc:1467] Processing CreateTablet for tablet fb1848b778274b788853fcb99a639b88 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 14:05:54.746033 23388 tablet_bootstrap.cc:492] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:54.746709 23389 tablet_bootstrap.cc:492] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:54.754571 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cf069882cc4e469cd068f6da62ecbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.735819 23255 tablet_service.cc:1467] Processing CreateTablet for tablet cc8b4f12c6d843efbcc7359b62a52cf5 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 14:05:54.762490 23258 tablet_service.cc:1467] Processing CreateTablet for tablet fb1848b778274b788853fcb99a639b88 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 14:05:54.762758 23337 tablet_service.cc:1467] Processing CreateTablet for tablet 994cb5cd6fe74f2293ca092bc550ffb3 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 14:05:54.763940 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994cb5cd6fe74f2293ca092bc550ffb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.765926 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 01e5a9e6ff8641eead910a9416773d5f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 14:05:54.767064 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e5a9e6ff8641eead910a9416773d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.757922 23182 tablet_service.cc:1467] Processing CreateTablet for tablet cc8b4f12c6d843efbcc7359b62a52cf5 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 14:05:54.757994 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5b811e321d4679bb9a64f5965c6a7d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 14:05:54.781555 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 69a22b60b6bb46189f2c845b2d299a3f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 14:05:54.782716 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a22b60b6bb46189f2c845b2d299a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.783043 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1848b778274b788853fcb99a639b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.783365 23336 tablet_service.cc:1467] Processing CreateTablet for tablet 69a6ce78fbc0428199f226fd9120217f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 14:05:54.783959 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8b4f12c6d843efbcc7359b62a52cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.784476 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a6ce78fbc0428199f226fd9120217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.784535 23331 tablet_service.cc:1467] Processing CreateTablet for tablet 3f93998b6cbe40bc9dd530d7f03e8cbf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 14:05:54.787065 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5b811e321d4679bb9a64f5965c6a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.790616 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1848b778274b788853fcb99a639b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.791437 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a156a5f03e4c8c97f11fd328ad5018. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.787923 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f93998b6cbe40bc9dd530d7f03e8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.800002 23253 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5b811e321d4679bb9a64f5965c6a7d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 14:05:54.804050 23391 ts_tablet_manager.cc:1398] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.140s user 0.005s sys 0.017s
I20240502 14:05:54.807554 23222 heartbeater.cc:503] Master 127.21.129.62:37293 was elected leader, sending a full tablet report...
I20240502 14:05:54.812104 23257 tablet_service.cc:1467] Processing CreateTablet for tablet b90f20f57cf34057a5e3301a044ad3ca (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 14:05:54.814407 23391 tablet_bootstrap.cc:492] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:54.825126 23389 tablet_bootstrap.cc:654] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:54.824786 23328 tablet_service.cc:1467] Processing CreateTablet for tablet 8b17aa98c3f04a58a5613d8b4af43f2e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 14:05:54.827641 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b17aa98c3f04a58a5613d8b4af43f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.832154 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8b4f12c6d843efbcc7359b62a52cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.832834 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5b811e321d4679bb9a64f5965c6a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.819052 23183 tablet_service.cc:1467] Processing CreateTablet for tablet b90f20f57cf34057a5e3301a044ad3ca (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 14:05:54.836344 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90f20f57cf34057a5e3301a044ad3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.836449 23332 tablet_service.cc:1467] Processing CreateTablet for tablet 69461a11cb2240929c388c86126ad78c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 14:05:54.837620 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69461a11cb2240929c388c86126ad78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.839661 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90f20f57cf34057a5e3301a044ad3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.841454 23184 tablet_service.cc:1467] Processing CreateTablet for tablet 69a6ce78fbc0428199f226fd9120217f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 14:05:54.843374 23330 tablet_service.cc:1467] Processing CreateTablet for tablet 6c123d564235427795f0271d9ae4213c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 14:05:54.844516 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c123d564235427795f0271d9ae4213c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.844775 23333 tablet_service.cc:1467] Processing CreateTablet for tablet 50b2b38a668947049fc57d84a48e6bdb (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 14:05:54.845176 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 994cb5cd6fe74f2293ca092bc550ffb3 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 14:05:54.846053 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b2b38a668947049fc57d84a48e6bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.846269 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994cb5cd6fe74f2293ca092bc550ffb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.848021 23261 tablet_service.cc:1467] Processing CreateTablet for tablet 69a6ce78fbc0428199f226fd9120217f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 14:05:54.849154 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a6ce78fbc0428199f226fd9120217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.854209 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a6ce78fbc0428199f226fd9120217f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.869194 23391 tablet_bootstrap.cc:654] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:54.879354 23388 tablet_bootstrap.cc:654] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:54.881806 23337 tablet_service.cc:1467] Processing CreateTablet for tablet 25d41b81c5154af7a4644e29408cb4cf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 14:05:54.882947 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d41b81c5154af7a4644e29408cb4cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.884761 23393 raft_consensus.cc:482] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:54.885237 23393 raft_consensus.cc:504] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.889643 23335 tablet_service.cc:1467] Processing CreateTablet for tablet cd50bda89ed043c4900bd94025639e77 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 14:05:54.906860 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd50bda89ed043c4900bd94025639e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.920533 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 994cb5cd6fe74f2293ca092bc550ffb3 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 14:05:54.921977 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994cb5cd6fe74f2293ca092bc550ffb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.951004 23396 raft_consensus.cc:482] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:54.951474 23396 raft_consensus.cc:504] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.953601 23396 leader_election.cc:288] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:54.953356 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 01e5a9e6ff8641eead910a9416773d5f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 14:05:54.959800 23393 leader_election.cc:288] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:54.978497 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 69a22b60b6bb46189f2c845b2d299a3f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 14:05:54.995489 23388 tablet_bootstrap.cc:492] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:54.995935 23388 ts_tablet_manager.cc:1367] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.255s user 0.016s sys 0.004s
I20240502 14:05:54.997130 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 01e5a9e6ff8641eead910a9416773d5f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 14:05:54.998304 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e5a9e6ff8641eead910a9416773d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:54.998569 23388 raft_consensus.cc:348] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:54.951481 23182 tablet_service.cc:1467] Processing CreateTablet for tablet 3f93998b6cbe40bc9dd530d7f03e8cbf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 14:05:55.022231 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 50b2b38a668947049fc57d84a48e6bdb (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 14:05:55.023290 23388 raft_consensus.cc:374] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.023623 23388 raft_consensus.cc:729] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.024281 23388 consensus_queue.cc:260] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.026036 23178 tablet_service.cc:1467] Processing CreateTablet for tablet 8b17aa98c3f04a58a5613d8b4af43f2e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 14:05:54.988595 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e5a9e6ff8641eead910a9416773d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.032230 23388 ts_tablet_manager.cc:1398] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.036s user 0.004s sys 0.003s
I20240502 14:05:55.033022 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f93998b6cbe40bc9dd530d7f03e8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.038362 23328 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc8a96cb68549b2b10d5d759fb49b86 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 14:05:55.039225 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a22b60b6bb46189f2c845b2d299a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.042600 23334 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2a73e349254db1a61fc140ca00d99c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 14:05:55.043155 23257 tablet_service.cc:1467] Processing CreateTablet for tablet 69461a11cb2240929c388c86126ad78c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 14:05:54.991030 23259 tablet_service.cc:1467] Processing CreateTablet for tablet 3f93998b6cbe40bc9dd530d7f03e8cbf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20240502 14:05:55.047333 23079 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 44814.8us
I20240502 14:05:55.026149 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 50b2b38a668947049fc57d84a48e6bdb (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 14:05:55.049916 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2a73e349254db1a61fc140ca00d99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.051230 23389 tablet_bootstrap.cc:492] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.069746 23389 ts_tablet_manager.cc:1367] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.328s user 0.023s sys 0.000s
I20240502 14:05:55.072276 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc8a96cb68549b2b10d5d759fb49b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.057006 23184 tablet_service.cc:1467] Processing CreateTablet for tablet 6c123d564235427795f0271d9ae4213c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 14:05:55.048205 23391 tablet_bootstrap.cc:492] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.059728 23262 tablet_service.cc:1467] Processing CreateTablet for tablet 8b17aa98c3f04a58a5613d8b4af43f2e (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 14:05:55.061849 23181 tablet_service.cc:1467] Processing CreateTablet for tablet 69461a11cb2240929c388c86126ad78c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 14:05:55.062326 23330 tablet_service.cc:1467] Processing CreateTablet for tablet 6adefff536c04350a45b88a896f35226 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 14:05:55.068037 23388 tablet_bootstrap.cc:492] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.062094 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 55c964342c58447b955034291dc40cb6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 14:05:55.084342 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c964342c58447b955034291dc40cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.087458 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6adefff536c04350a45b88a896f35226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.087927 23391 ts_tablet_manager.cc:1367] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.274s user 0.011s sys 0.006s
I20240502 14:05:55.092079 23389 raft_consensus.cc:348] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.092839 23389 raft_consensus.cc:374] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.093120 23389 raft_consensus.cc:729] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.093403 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 6c123d564235427795f0271d9ae4213c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 14:05:55.093820 23389 consensus_queue.cc:260] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.094998 23185 tablet_service.cc:1467] Processing CreateTablet for tablet 69a22b60b6bb46189f2c845b2d299a3f (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 14:05:55.098125 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee3ba4c1e8843d8b94029ece423580f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
W20240502 14:05:55.099947 23310 leader_election.cc:341] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.100689 23253 tablet_service.cc:1467] Processing CreateTablet for tablet 25d41b81c5154af7a4644e29408cb4cf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 14:05:55.105079 23336 tablet_service.cc:1467] Processing CreateTablet for tablet 6132f4449de74e15a3cbe93f03a74b71 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 14:05:55.105330 23337 tablet_service.cc:1467] Processing CreateTablet for tablet afca5b05bac0423ca02254a903b07eca (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 14:05:55.106480 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6132f4449de74e15a3cbe93f03a74b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.107069 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afca5b05bac0423ca02254a903b07eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.090783 23391 raft_consensus.cc:348] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.115440 23391 raft_consensus.cc:374] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.115808 23391 raft_consensus.cc:729] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.116582 23391 consensus_queue.cc:260] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.117615 23328 tablet_service.cc:1467] Processing CreateTablet for tablet fbb27a4bdc15414cb260ffc2b673b5ab (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 14:05:55.118767 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb27a4bdc15414cb260ffc2b673b5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.119277 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b17aa98c3f04a58a5613d8b4af43f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.119410 23395 raft_consensus.cc:482] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.119863 23395 raft_consensus.cc:504] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.122282 23395 leader_election.cc:288] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179), cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:05:55.126368 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c123d564235427795f0271d9ae4213c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.127596 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69461a11cb2240929c388c86126ad78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.130007 23335 tablet_service.cc:1467] Processing CreateTablet for tablet 11d15424bb7b49cb899aba05cea68af0 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 14:05:55.118578 23179 tablet_service.cc:1467] Processing CreateTablet for tablet cd50bda89ed043c4900bd94025639e77 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 14:05:55.124138 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b2b38a668947049fc57d84a48e6bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.134770 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a22b60b6bb46189f2c845b2d299a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.135342 23396 raft_consensus.cc:482] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.146701 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b2b38a668947049fc57d84a48e6bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.159055 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd50bda89ed043c4900bd94025639e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.146080 23396 raft_consensus.cc:504] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.162803 23396 leader_election.cc:288] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.167428 23391 ts_tablet_manager.cc:1398] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.079s user 0.007s sys 0.000s
I20240502 14:05:55.168399 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d15424bb7b49cb899aba05cea68af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.171314 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d41b81c5154af7a4644e29408cb4cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.173121 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b17aa98c3f04a58a5613d8b4af43f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.175427 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69461a11cb2240929c388c86126ad78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.179493 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f93998b6cbe40bc9dd530d7f03e8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.187776 23389 ts_tablet_manager.cc:1398] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
I20240502 14:05:55.188819 23261 tablet_service.cc:1467] Processing CreateTablet for tablet cd50bda89ed043c4900bd94025639e77 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 14:05:55.190018 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd50bda89ed043c4900bd94025639e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.192724 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc8a96cb68549b2b10d5d759fb49b86 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 14:05:55.193755 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c123d564235427795f0271d9ae4213c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.201927 23332 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5d4ce1413141d0a1a08c3397a8ea70 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 14:05:55.202211 23395 raft_consensus.cc:482] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.203119 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5d4ce1413141d0a1a08c3397a8ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.202693 23395 raft_consensus.cc:504] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.202234 23255 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2a73e349254db1a61fc140ca00d99c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 14:05:55.204444 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2a73e349254db1a61fc140ca00d99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.204651 23395 leader_election.cc:288] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.168360 23391 tablet_bootstrap.cc:492] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.188764 23389 tablet_bootstrap.cc:492] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.220717 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 0b074ab1c0a041dabe1f6243776950a5 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 14:05:55.221846 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b074ab1c0a041dabe1f6243776950a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.213953 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc8a96cb68549b2b10d5d759fb49b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.229141 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 55c964342c58447b955034291dc40cb6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 14:05:55.239663 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee3ba4c1e8843d8b94029ece423580f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:55.241117 23313 leader_election.cc:341] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.241550 23313 leader_election.cc:302] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.242589 23393 raft_consensus.cc:2726] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:55.243252 23331 tablet_service.cc:1467] Processing CreateTablet for tablet 1eda455829f14de7865c64312c2bdb9d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 14:05:55.244431 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eda455829f14de7865c64312c2bdb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.244313 23335 tablet_service.cc:1467] Processing CreateTablet for tablet ec446373869d4c05ad8c5807b73740bf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 14:05:55.248800 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec446373869d4c05ad8c5807b73740bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.253650 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2a73e349254db1a61fc140ca00d99c (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 14:05:55.246628 23183 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc8a96cb68549b2b10d5d759fb49b86 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 14:05:55.254819 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2a73e349254db1a61fc140ca00d99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.255582 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc8a96cb68549b2b10d5d759fb49b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.208518 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 25d41b81c5154af7a4644e29408cb4cf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 14:05:55.262732 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d41b81c5154af7a4644e29408cb4cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.270892 23181 tablet_service.cc:1467] Processing CreateTablet for tablet 55c964342c58447b955034291dc40cb6 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 14:05:55.279453 23333 tablet_service.cc:1467] Processing CreateTablet for tablet ff7e187bdb41455fbddec2888eb6efa8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 14:05:55.280496 23391 tablet_bootstrap.cc:654] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.284027 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 6adefff536c04350a45b88a896f35226 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 14:05:55.284791 23389 tablet_bootstrap.cc:654] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.285212 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6adefff536c04350a45b88a896f35226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.289705 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 6adefff536c04350a45b88a896f35226 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 14:05:55.290196 23184 tablet_service.cc:1467] Processing CreateTablet for tablet 6132f4449de74e15a3cbe93f03a74b71 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 14:05:55.292572 23388 tablet_bootstrap.cc:654] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.294102 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.272032 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c964342c58447b955034291dc40cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.280622 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e187bdb41455fbddec2888eb6efa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.246783 23334 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0ebc495868467d95f7e6b636732483 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20240502 14:05:55.299443 23159 leader_election.cc:341] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.300590 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e4ee45e707479b874fab4903df88f6" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.301211 23273 raft_consensus.cc:2454] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:55.302361 23159 leader_election.cc:302] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:55.303179 23396 raft_consensus.cc:2781] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.303630 23396 raft_consensus.cc:482] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.303941 23396 raft_consensus.cc:3037] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.305570 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ebc495868467d95f7e6b636732483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.308593 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:05:55.310798 23162 leader_election.cc:341] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.310052 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e4ee45e707479b874fab4903df88f6" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.312304 23162 leader_election.cc:302] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:55.309988 23257 tablet_service.cc:1467] Processing CreateTablet for tablet 6132f4449de74e15a3cbe93f03a74b71 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 14:05:55.230888 23337 tablet_service.cc:1467] Processing CreateTablet for tablet fa7635b879734a9e9c6a7f8c154251f1 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 14:05:55.319447 23414 raft_consensus.cc:2726] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:05:55.319550 23162 leader_election.cc:341] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.324054 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c964342c58447b955034291dc40cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.331653 23179 tablet_service.cc:1467] Processing CreateTablet for tablet afca5b05bac0423ca02254a903b07eca (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 14:05:55.333685 23259 tablet_service.cc:1467] Processing CreateTablet for tablet afca5b05bac0423ca02254a903b07eca (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 14:05:55.335065 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afca5b05bac0423ca02254a903b07eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.337374 23260 tablet_service.cc:1467] Processing CreateTablet for tablet fbb27a4bdc15414cb260ffc2b673b5ab (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 14:05:55.338963 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb27a4bdc15414cb260ffc2b673b5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.342535 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7635b879734a9e9c6a7f8c154251f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.348820 23183 tablet_service.cc:1467] Processing CreateTablet for tablet fbb27a4bdc15414cb260ffc2b673b5ab (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 14:05:55.353101 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6132f4449de74e15a3cbe93f03a74b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.354470 23262 tablet_service.cc:1467] Processing CreateTablet for tablet 1eda455829f14de7865c64312c2bdb9d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 14:05:55.355806 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eda455829f14de7865c64312c2bdb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.366196 23396 raft_consensus.cc:504] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.371047 23182 tablet_service.cc:1467] Processing CreateTablet for tablet 1eda455829f14de7865c64312c2bdb9d (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 14:05:55.379549 23255 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5d4ce1413141d0a1a08c3397a8ea70 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 14:05:55.380779 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5d4ce1413141d0a1a08c3397a8ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.381348 23253 tablet_service.cc:1467] Processing CreateTablet for tablet 0b074ab1c0a041dabe1f6243776950a5 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 14:05:55.389377 23389 tablet_bootstrap.cc:492] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.384410 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6132f4449de74e15a3cbe93f03a74b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.389834 23389 ts_tablet_manager.cc:1367] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.201s user 0.007s sys 0.010s
I20240502 14:05:55.391479 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6adefff536c04350a45b88a896f35226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.392817 23396 leader_election.cc:288] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.393483 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e4ee45e707479b874fab4903df88f6" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:55.393802 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e4ee45e707479b874fab4903df88f6" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:55.394075 23273 raft_consensus.cc:3037] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:55.396525 23162 leader_election.cc:341] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.398213 23178 tablet_service.cc:1467] Processing CreateTablet for tablet 11d15424bb7b49cb899aba05cea68af0 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 14:05:55.404894 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5d4ce1413141d0a1a08c3397a8ea70 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 14:05:55.409564 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 0b074ab1c0a041dabe1f6243776950a5 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 14:05:55.411424 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afca5b05bac0423ca02254a903b07eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.412005 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb27a4bdc15414cb260ffc2b673b5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.414954 23273 raft_consensus.cc:2454] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:55.417891 23257 tablet_service.cc:1467] Processing CreateTablet for tablet 11d15424bb7b49cb899aba05cea68af0 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 14:05:55.418761 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eda455829f14de7865c64312c2bdb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.418654 23159 leader_election.cc:302] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:55.420796 23396 raft_consensus.cc:2781] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:55.421348 23396 raft_consensus.cc:686] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:05:55.422061 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e4ee45e707479b874fab4903df88f6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.420714 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b50f554edf4267b72624ec5b865c2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.423168 23346 raft_consensus.cc:2454] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
W20240502 14:05:55.423739 23238 leader_election.cc:341] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.422119 23396 consensus_queue.cc:237] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.424527 23238 leader_election.cc:302] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:55.425307 23395 raft_consensus.cc:2781] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.425659 23395 raft_consensus.cc:482] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.425992 23395 raft_consensus.cc:3037] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.426179 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5d4ce1413141d0a1a08c3397a8ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.432915 23256 tablet_service.cc:1467] Processing CreateTablet for tablet fa7635b879734a9e9c6a7f8c154251f1 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 14:05:55.436539 23185 tablet_service.cc:1467] Processing CreateTablet for tablet fa7635b879734a9e9c6a7f8c154251f1 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 14:05:55.437508 23395 raft_consensus.cc:504] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.443544 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7635b879734a9e9c6a7f8c154251f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.444325 23388 tablet_bootstrap.cc:492] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.444788 23388 ts_tablet_manager.cc:1367] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.395s user 0.025s sys 0.048s
I20240502 14:05:55.447456 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b074ab1c0a041dabe1f6243776950a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.447988 23391 tablet_bootstrap.cc:492] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.447500 23388 raft_consensus.cc:348] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.448449 23391 ts_tablet_manager.cc:1367] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.280s user 0.008s sys 0.007s
I20240502 14:05:55.448488 23388 raft_consensus.cc:374] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.449107 23388 raft_consensus.cc:729] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.450080 23388 consensus_queue.cc:260] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.455227 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d15424bb7b49cb899aba05cea68af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.454793 23254 tablet_service.cc:1467] Processing CreateTablet for tablet ff7e187bdb41455fbddec2888eb6efa8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 14:05:55.441767 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b074ab1c0a041dabe1f6243776950a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.456758 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0ebc495868467d95f7e6b636732483 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 14:05:55.458245 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b50f554edf4267b72624ec5b865c2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:55.465099 23346 raft_consensus.cc:3037] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.440387 23389 raft_consensus.cc:348] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.467916 23389 raft_consensus.cc:374] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.468240 23389 raft_consensus.cc:729] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.468981 23389 consensus_queue.cc:260] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.471403 23389 ts_tablet_manager.cc:1398] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.081s user 0.006s sys 0.002s
I20240502 14:05:55.459327 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b50f554edf4267b72624ec5b865c2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.460178 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e4ee45e707479b874fab4903df88f6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.460996 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b50f554edf4267b72624ec5b865c2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:55.471511 23346 raft_consensus.cc:2454] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:55.457358 23180 tablet_service.cc:1467] Processing CreateTablet for tablet ec446373869d4c05ad8c5807b73740bf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20240502 14:05:55.492388 23238 leader_election.cc:341] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.494052 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec446373869d4c05ad8c5807b73740bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:05:55.494263 23238 leader_election.cc:341] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.494611 23388 ts_tablet_manager.cc:1398] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.050s user 0.007s sys 0.005s
I20240502 14:05:55.494992 23238 leader_election.cc:302] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.495517 23388 tablet_bootstrap.cc:492] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.472501 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ebc495868467d95f7e6b636732483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.440802 23395 leader_election.cc:288] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179), cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:05:55.478044 23255 tablet_service.cc:1467] Processing CreateTablet for tablet ec446373869d4c05ad8c5807b73740bf (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 14:05:55.480147 23185 tablet_service.cc:1467] Processing CreateTablet for tablet ff7e187bdb41455fbddec2888eb6efa8 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 14:05:55.481753 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0ebc495868467d95f7e6b636732483 (DEFAULT_TABLE table=test-workload [id=b135b2923733409a98bf07d2863419d5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 14:05:55.489742 23091 catalog_manager.cc:5484] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:55.497623 23389 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.499713 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d15424bb7b49cb899aba05cea68af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.499802 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e187bdb41455fbddec2888eb6efa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.451130 23391 raft_consensus.cc:348] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.508253 23391 raft_consensus.cc:374] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.508555 23391 raft_consensus.cc:729] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.509217 23391 consensus_queue.cc:260] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.516124 23416 raft_consensus.cc:2726] T 35e4ee45e707479b874fab4903df88f6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:55.520859 23389 tablet_bootstrap.cc:654] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.526132 23260 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 50) took 1020 ms. Trace:
I20240502 14:05:55.516342 23261 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 43) took 1074 ms. Trace:
I20240502 14:05:55.527595 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec446373869d4c05ad8c5807b73740bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.516026 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ebc495868467d95f7e6b636732483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.526877 23261 rpcz_store.cc:276] 0502 14:05:54.457871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.187981 (+730110us) service_pool.cc:225] Handling call
0502 14:05:55.189598 (+ 1617us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.189688 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.416562 (+226874us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.496776 (+ 80214us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.516272 (+ 19496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.511477 23391 ts_tablet_manager.cc:1398] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20240502 14:05:55.536001 23391 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.543345 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7635b879734a9e9c6a7f8c154251f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.544613 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 48) took 1047 ms. Trace:
I20240502 14:05:55.544981 23184 rpcz_store.cc:276] 0502 14:05:54.497001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.289397 (+792396us) service_pool.cc:225] Handling call
0502 14:05:55.290936 (+ 1539us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.291027 (+ 91us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.475708 (+184681us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.544104 (+ 68396us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.544551 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.548995 23179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 49) took 1051 ms. Trace:
I20240502 14:05:55.549363 23179 rpcz_store.cc:276] 0502 14:05:54.497340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.296683 (+799343us) service_pool.cc:225] Handling call
0502 14:05:55.332463 (+ 35780us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.332669 (+ 206us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.487179 (+154510us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.548551 (+ 61372us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.548939 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.550467 23262 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 51) took 1044 ms. Trace:
I20240502 14:05:55.555627 23337 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56144 (request call id 54) took 1070 ms. Trace:
I20240502 14:05:55.555986 23337 rpcz_store.cc:276] 0502 14:05:54.485100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.230088 (+744988us) service_pool.cc:225] Handling call
0502 14:05:55.325953 (+ 95865us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.330899 (+ 4946us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.465845 (+134946us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.555129 (+ 89284us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.555570 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.558138 23180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 56) took 1027 ms. Trace:
I20240502 14:05:55.558400 23180 rpcz_store.cc:276] 0502 14:05:54.530755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.456576 (+925821us) service_pool.cc:225] Handling call
0502 14:05:55.493624 (+ 37048us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.493720 (+ 96us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.502695 (+ 8975us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.557769 (+ 55074us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.558089 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.562991 23183 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 50) took 1065 ms. Trace:
I20240502 14:05:55.563515 23183 rpcz_store.cc:276] 0502 14:05:54.497654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.347918 (+850264us) service_pool.cc:225] Handling call
0502 14:05:55.350934 (+ 3016us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.351265 (+ 331us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.423152 (+ 71887us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.562446 (+139294us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.562870 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.564244 23181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 46) took 1068 ms. Trace:
I20240502 14:05:55.564641 23181 rpcz_store.cc:276] 0502 14:05:54.496263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.270071 (+773808us) service_pool.cc:225] Handling call
0502 14:05:55.271640 (+ 1569us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.271715 (+ 75us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.501392 (+229677us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.563776 (+ 62384us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.564181 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.545292 23334 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56144 (request call id 58) took 1058 ms. Trace:
I20240502 14:05:55.550987 23258 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 58) took 1023 ms. Trace:
I20240502 14:05:55.565655 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e187bdb41455fbddec2888eb6efa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:55.575659 23388 tablet_bootstrap.cc:654] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.539754 23260 rpcz_store.cc:276] 0502 14:05:54.506036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.330768 (+824732us) service_pool.cc:225] Handling call
0502 14:05:55.338529 (+ 7761us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.338619 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.461465 (+122846us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.525703 (+ 64238us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.526076 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.569180 23334 rpcz_store.cc:276] 0502 14:05:54.486785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.246024 (+759239us) service_pool.cc:225] Handling call
0502 14:05:55.299535 (+ 53511us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.299620 (+ 85us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.473493 (+173873us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.544842 (+ 71349us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.545231 (+ 389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.582656 23253 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 53) took 1075 ms. Trace:
I20240502 14:05:55.583315 23253 rpcz_store.cc:276] 0502 14:05:54.507145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.379915 (+872770us) service_pool.cc:225] Handling call
0502 14:05:55.382535 (+ 2620us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.387787 (+ 5252us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.542962 (+155175us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.582126 (+ 39164us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.582596 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.550825 23262 rpcz_store.cc:276] 0502 14:05:54.506372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.353553 (+847181us) service_pool.cc:225] Handling call
0502 14:05:55.355337 (+ 1784us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.355439 (+ 102us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.440738 (+ 85299us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.499269 (+ 58531us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.550408 (+ 51139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.568820 23258 rpcz_store.cc:276] 0502 14:05:54.528012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.455959 (+927947us) service_pool.cc:225] Handling call
0502 14:05:55.472086 (+ 16127us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.472172 (+ 86us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.505743 (+ 33571us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.542889 (+ 37146us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.550931 (+ 8042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.595922 23182 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 51) took 1098 ms. Trace:
I20240502 14:05:55.596319 23182 rpcz_store.cc:276] 0502 14:05:54.497987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.370160 (+872173us) service_pool.cc:225] Handling call
0502 14:05:55.403258 (+ 33098us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.403356 (+ 98us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.527083 (+123727us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.595418 (+ 68335us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.595852 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8064}
I20240502 14:05:55.599745 23185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 57) took 1068 ms. Trace:
I20240502 14:05:55.600124 23185 rpcz_store.cc:276] 0502 14:05:54.531119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.479311 (+948192us) service_pool.cc:225] Handling call
0502 14:05:55.499381 (+ 20070us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.499470 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.591656 (+ 92186us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.599320 (+ 7664us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.599688 (+ 368us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 14:05:55.591720 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 96804.4us
I20240502 14:05:55.596825 23177 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 58) took 1065 ms. Trace:
I20240502 14:05:55.605111 23238 leader_election.cc:302] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.604897 23177 rpcz_store.cc:276] 0502 14:05:54.531504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.480836 (+949332us) service_pool.cc:225] Handling call
0502 14:05:55.507469 (+ 26633us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.515669 (+ 8200us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.556883 (+ 41214us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.596392 (+ 39509us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.596762 (+ 370us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.605993 23418 raft_consensus.cc:2781] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:55.606389 23418 raft_consensus.cc:686] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:55.607131 23418 consensus_queue.cc:237] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.608206 23391 tablet_bootstrap.cc:654] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.608881 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40056 (request call id 47) took 1112 ms. Trace:
I20240502 14:05:55.612185 23389 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.612658 23389 ts_tablet_manager.cc:1367] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.009s
I20240502 14:05:55.615096 23254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 57) took 1087 ms. Trace:
I20240502 14:05:55.615469 23254 rpcz_store.cc:276] 0502 14:05:54.527717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.453918 (+926201us) service_pool.cc:225] Handling call
0502 14:05:55.462885 (+ 8967us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.462970 (+ 85us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.577671 (+114701us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.614652 (+ 36981us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.615043 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.617817 23388 tablet_bootstrap.cc:492] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.618252 23388 ts_tablet_manager.cc:1367] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.123s user 0.014s sys 0.001s
I20240502 14:05:55.620870 23255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 56) took 1093 ms. Trace:
I20240502 14:05:55.621197 23255 rpcz_store.cc:276] 0502 14:05:54.527395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.426344 (+898949us) service_pool.cc:225] Handling call
0502 14:05:55.508847 (+ 82503us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.508960 (+ 113us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.551023 (+ 42063us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.620529 (+ 69506us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.620818 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.619247 23257 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 55) took 1092 ms. Trace:
I20240502 14:05:55.622503 23257 rpcz_store.cc:276] 0502 14:05:54.527053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.417106 (+890053us) service_pool.cc:225] Handling call
0502 14:05:55.418606 (+ 1500us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.418699 (+ 93us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.600273 (+181574us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.618759 (+ 18486us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.619168 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.624222 23388 raft_consensus.cc:348] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.624740 23388 raft_consensus.cc:374] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.625027 23389 raft_consensus.cc:348] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.625751 23389 raft_consensus.cc:374] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.624969 23388 raft_consensus.cc:729] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.626056 23389 raft_consensus.cc:729] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.627245 23389 consensus_queue.cc:260] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.627692 23090 catalog_manager.cc:5484] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:55.628715 23388 consensus_queue.cc:260] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.609244 23186 rpcz_store.cc:276] 0502 14:05:54.496670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.288884 (+792214us) service_pool.cc:225] Handling call
0502 14:05:55.317965 (+ 29081us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.323133 (+ 5168us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.505287 (+182154us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.608457 (+103170us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.608820 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.626073 23256 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52056 (request call id 54) took 1099 ms. Trace:
I20240502 14:05:55.633885 23256 rpcz_store.cc:276] 0502 14:05:54.526710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:05:55.394858 (+868148us) service_pool.cc:225] Handling call
0502 14:05:55.433684 (+ 38826us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 14:05:55.433767 (+ 83us) ts_tablet_manager.cc:655] Creating new metadata...
0502 14:05:55.597984 (+164217us) tablet_metadata.cc:641] Metadata flushed
0502 14:05:55.625638 (+ 27654us) tablet_replica.cc:205] Creating consensus instance
0502 14:05:55.626021 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 14:05:55.635131 23389 ts_tablet_manager.cc:1398] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.022s user 0.003s sys 0.004s
I20240502 14:05:55.631170 23388 ts_tablet_manager.cc:1398] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.013s user 0.002s sys 0.006s
I20240502 14:05:55.636440 23389 tablet_bootstrap.cc:492] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.636564 23388 tablet_bootstrap.cc:492] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.640450 23391 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.640895 23391 ts_tablet_manager.cc:1367] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.011s
I20240502 14:05:55.641831 23389 tablet_bootstrap.cc:654] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.642418 23388 tablet_bootstrap.cc:654] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.643396 23391 raft_consensus.cc:348] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.644101 23391 raft_consensus.cc:374] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.644623 23391 raft_consensus.cc:729] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.645659 23391 consensus_queue.cc:260] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.648295 23391 ts_tablet_manager.cc:1398] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240502 14:05:55.649209 23388 tablet_bootstrap.cc:492] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.649302 23391 tablet_bootstrap.cc:492] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.649523 23388 ts_tablet_manager.cc:1367] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240502 14:05:55.655411 23388 raft_consensus.cc:348] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.656086 23388 raft_consensus.cc:374] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.656373 23388 raft_consensus.cc:729] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.656847 23391 tablet_bootstrap.cc:654] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.657447 23389 tablet_bootstrap.cc:492] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.656996 23388 consensus_queue.cc:260] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.657908 23389 ts_tablet_manager.cc:1367] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.009s
I20240502 14:05:55.659288 23388 ts_tablet_manager.cc:1398] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240502 14:05:55.660322 23388 tablet_bootstrap.cc:492] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.660528 23389 raft_consensus.cc:348] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.661276 23389 raft_consensus.cc:374] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.661567 23389 raft_consensus.cc:729] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.663697 23389 consensus_queue.cc:260] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.665763 23388 tablet_bootstrap.cc:654] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.668776 23389 ts_tablet_manager.cc:1398] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240502 14:05:55.669770 23389 tablet_bootstrap.cc:492] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.675830 23391 tablet_bootstrap.cc:492] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.676316 23391 ts_tablet_manager.cc:1367] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.010s
I20240502 14:05:55.676467 23389 tablet_bootstrap.cc:654] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.679023 23391 raft_consensus.cc:348] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.679869 23391 raft_consensus.cc:374] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.680209 23391 raft_consensus.cc:729] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.681031 23391 consensus_queue.cc:260] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:55.684767 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:05:55.689806 23391 ts_tablet_manager.cc:1398] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20240502 14:05:55.690651 23391 tablet_bootstrap.cc:492] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
W20240502 14:05:55.695290 23299 tablet.cc:2369] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:55.695384 23389 tablet_bootstrap.cc:492] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.695943 23389 ts_tablet_manager.cc:1367] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.008s
I20240502 14:05:55.699033 23391 tablet_bootstrap.cc:654] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.698845 23389 raft_consensus.cc:348] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.699841 23389 raft_consensus.cc:374] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.700124 23389 raft_consensus.cc:729] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.700740 23389 consensus_queue.cc:260] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.702091 23388 tablet_bootstrap.cc:492] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.702617 23388 ts_tablet_manager.cc:1367] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.043s user 0.021s sys 0.021s
I20240502 14:05:55.702893 23389 ts_tablet_manager.cc:1398] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 14:05:55.704067 23389 tablet_bootstrap.cc:492] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.705394 23388 raft_consensus.cc:348] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.706152 23388 raft_consensus.cc:374] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.706475 23388 raft_consensus.cc:729] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.707283 23388 consensus_queue.cc:260] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.709684 23388 ts_tablet_manager.cc:1398] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240502 14:05:55.712220 23389 tablet_bootstrap.cc:654] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.712791 23388 tablet_bootstrap.cc:492] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.718647 23388 tablet_bootstrap.cc:654] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.724056 23391 tablet_bootstrap.cc:492] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.724531 23391 ts_tablet_manager.cc:1367] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.016s
I20240502 14:05:55.724573 23389 tablet_bootstrap.cc:492] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.724820 23388 tablet_bootstrap.cc:492] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.724790 23418 raft_consensus.cc:482] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.725189 23389 ts_tablet_manager.cc:1367] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.011s
I20240502 14:05:55.725433 23418 raft_consensus.cc:504] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.726998 23391 raft_consensus.cc:348] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.727384 23418 leader_election.cc:288] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.727747 23391 raft_consensus.cc:374] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.725425 23388 ts_tablet_manager.cc:1367] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 14:05:55.728225 23391 raft_consensus.cc:729] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.727793 23389 raft_consensus.cc:348] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.728704 23389 raft_consensus.cc:374] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.728852 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009254b7af044911807829cd2423373e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.729102 23389 raft_consensus.cc:729] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.728907 23391 consensus_queue.cc:260] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.729633 23196 raft_consensus.cc:2454] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:55.729902 23389 consensus_queue.cc:260] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.730294 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009254b7af044911807829cd2423373e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.730898 23346 raft_consensus.cc:2454] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:55.730558 23388 raft_consensus.cc:348] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.731441 23388 raft_consensus.cc:374] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.731796 23388 raft_consensus.cc:729] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.732403 23238 leader_election.cc:302] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:55.732511 23388 consensus_queue.cc:260] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.733610 23391 ts_tablet_manager.cc:1398] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20240502 14:05:55.734519 23391 tablet_bootstrap.cc:492] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.733203 23418 raft_consensus.cc:2781] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.734939 23388 ts_tablet_manager.cc:1398] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240502 14:05:55.735073 23418 raft_consensus.cc:482] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.735565 23418 raft_consensus.cc:3037] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.735935 23388 tablet_bootstrap.cc:492] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.738699 23389 ts_tablet_manager.cc:1398] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:05:55.739624 23389 tablet_bootstrap.cc:492] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.739856 23391 tablet_bootstrap.cc:654] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.741722 23388 tablet_bootstrap.cc:654] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.744956 23389 tablet_bootstrap.cc:654] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.753003 23388 tablet_bootstrap.cc:492] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.753490 23388 ts_tablet_manager.cc:1367] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20240502 14:05:55.755908 23388 raft_consensus.cc:348] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.756573 23388 raft_consensus.cc:374] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.756827 23388 raft_consensus.cc:729] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.757611 23388 consensus_queue.cc:260] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.760465 23388 ts_tablet_manager.cc:1398] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240502 14:05:55.761117 23389 tablet_bootstrap.cc:492] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.761461 23389 ts_tablet_manager.cc:1367] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20240502 14:05:55.762214 23388 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.763852 23389 raft_consensus.cc:348] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.764498 23389 raft_consensus.cc:374] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.764746 23389 raft_consensus.cc:729] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.765363 23389 consensus_queue.cc:260] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.766736 23388 tablet_bootstrap.cc:654] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.767546 23389 ts_tablet_manager.cc:1398] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240502 14:05:55.767366 23418 raft_consensus.cc:504] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.769399 23393 raft_consensus.cc:482] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.769928 23393 raft_consensus.cc:504] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.771879 23393 leader_election.cc:288] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:55.768397 23389 tablet_bootstrap.cc:492] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.773793 23418 leader_election.cc:288] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.775705 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009254b7af044911807829cd2423373e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:55.776331 23196 raft_consensus.cc:3037] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.777508 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.778713 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.779875 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009254b7af044911807829cd2423373e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:55.780465 23346 raft_consensus.cc:3037] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:55.784046 23313 leader_election.cc:341] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:05:55.784788 23310 leader_election.cc:341] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.785199 23310 leader_election.cc:302] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.788434 23389 tablet_bootstrap.cc:654] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.791363 23393 raft_consensus.cc:2726] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:55.793020 23196 raft_consensus.cc:2454] T 009254b7af044911807829cd2423373e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:55.794195 23238 leader_election.cc:302] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:55.795081 23418 raft_consensus.cc:2781] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:55.795619 23418 raft_consensus.cc:686] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:55.796280 23346 raft_consensus.cc:2454] T 009254b7af044911807829cd2423373e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:55.796433 23418 consensus_queue.cc:237] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.800738 23388 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.801239 23388 ts_tablet_manager.cc:1367] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.016s
I20240502 14:05:55.803799 23388 raft_consensus.cc:348] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.804476 23388 raft_consensus.cc:374] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.804740 23388 raft_consensus.cc:729] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.805644 23388 consensus_queue.cc:260] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.807657 23391 tablet_bootstrap.cc:492] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.808149 23391 ts_tablet_manager.cc:1367] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.044s
I20240502 14:05:55.808434 23389 tablet_bootstrap.cc:492] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.808638 23388 ts_tablet_manager.cc:1398] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:05:55.808866 23389 ts_tablet_manager.cc:1367] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.005s
I20240502 14:05:55.809485 23388 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.810827 23391 raft_consensus.cc:348] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.811136 23389 raft_consensus.cc:348] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.811776 23391 raft_consensus.cc:374] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.811977 23389 raft_consensus.cc:374] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.812127 23391 raft_consensus.cc:729] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.812335 23389 raft_consensus.cc:729] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.813122 23391 consensus_queue.cc:260] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.813244 23389 consensus_queue.cc:260] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.814456 23388 tablet_bootstrap.cc:654] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.818481 23391 ts_tablet_manager.cc:1398] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240502 14:05:55.818488 23389 ts_tablet_manager.cc:1398] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:05:55.819617 23391 tablet_bootstrap.cc:492] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.819648 23389 tablet_bootstrap.cc:492] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.821177 23388 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.821573 23388 ts_tablet_manager.cc:1367] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240502 14:05:55.823889 23388 raft_consensus.cc:348] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.824520 23388 raft_consensus.cc:374] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.824774 23388 raft_consensus.cc:729] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.825160 23389 tablet_bootstrap.cc:654] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.825639 23391 tablet_bootstrap.cc:654] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.825431 23388 consensus_queue.cc:260] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.827845 23388 ts_tablet_manager.cc:1398] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:55.828845 23388 tablet_bootstrap.cc:492] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.833849 23388 tablet_bootstrap.cc:654] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.835937 23389 tablet_bootstrap.cc:492] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:55.836349 23389 ts_tablet_manager.cc:1367] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240502 14:05:55.837998 23393 raft_consensus.cc:482] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.838577 23388 tablet_bootstrap.cc:492] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.838440 23393 raft_consensus.cc:504] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.838987 23388 ts_tablet_manager.cc:1367] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:05:55.840513 23393 leader_election.cc:288] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:55.841106 23391 tablet_bootstrap.cc:492] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.840456 23091 catalog_manager.cc:5484] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:55.841531 23391 ts_tablet_manager.cc:1367] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20240502 14:05:55.841395 23388 raft_consensus.cc:348] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.842113 23388 raft_consensus.cc:374] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.842043 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e5363c783444bbb4f631a0ee208d9" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.842453 23388 raft_consensus.cc:729] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.849165 23388 consensus_queue.cc:260] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.849450 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e5363c783444bbb4f631a0ee208d9" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.850015 23391 raft_consensus.cc:348] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.850701 23391 raft_consensus.cc:374] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 14:05:55.850642 23313 leader_election.cc:341] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.851255 23391 raft_consensus.cc:729] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.851709 23389 raft_consensus.cc:348] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.852387 23389 raft_consensus.cc:374] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.851974 23391 consensus_queue.cc:260] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.852806 23273 raft_consensus.cc:2454] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:55.853878 23388 ts_tablet_manager.cc:1398] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:05:55.854532 23391 ts_tablet_manager.cc:1398] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20240502 14:05:55.854738 23388 tablet_bootstrap.cc:492] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.855459 23391 tablet_bootstrap.cc:492] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.855702 23310 leader_election.cc:302] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.856751 23393 raft_consensus.cc:2781] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.857105 23393 raft_consensus.cc:482] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.857421 23393 raft_consensus.cc:3037] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.860342 23388 tablet_bootstrap.cc:654] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.860920 23391 tablet_bootstrap.cc:654] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.864734 23389 raft_consensus.cc:729] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.865335 23396 raft_consensus.cc:482] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.865537 23389 consensus_queue.cc:260] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.865846 23396 raft_consensus.cc:504] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.867115 23393 raft_consensus.cc:504] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.867899 23396 leader_election.cc:288] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.870012 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5069e564364a3b82e55d410fc03cec" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.870924 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e5363c783444bbb4f631a0ee208d9" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:55.871210 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5069e564364a3b82e55d410fc03cec" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.871838 23272 raft_consensus.cc:2454] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:55.870904 23388 tablet_bootstrap.cc:492] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.872414 23388 ts_tablet_manager.cc:1367] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
W20240502 14:05:55.873143 23313 leader_election.cc:341] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.873531 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e5363c783444bbb4f631a0ee208d9" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:55.875043 23273 raft_consensus.cc:3037] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.872756 23159 leader_election.cc:302] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
W20240502 14:05:55.876200 23162 leader_election.cc:341] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.877830 23388 raft_consensus.cc:348] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.878489 23388 raft_consensus.cc:374] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.878748 23388 raft_consensus.cc:729] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.879333 23388 consensus_queue.cc:260] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.879979 23396 raft_consensus.cc:2781] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.880388 23396 raft_consensus.cc:482] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.880713 23396 raft_consensus.cc:3037] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.881443 23388 ts_tablet_manager.cc:1398] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:05:55.881426 23273 raft_consensus.cc:2454] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:55.882416 23388 tablet_bootstrap.cc:492] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:55.882550 23310 leader_election.cc:302] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.883845 23393 leader_election.cc:288] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:55.884609 23393 raft_consensus.cc:2781] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:55.885072 23393 raft_consensus.cc:686] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:55.885833 23393 consensus_queue.cc:237] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.887756 23388 tablet_bootstrap.cc:654] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.891791 23389 ts_tablet_manager.cc:1398] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.055s user 0.006s sys 0.002s
I20240502 14:05:55.892762 23389 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:55.896826 23414 consensus_queue.cc:1035] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:55.907505 23396 raft_consensus.cc:504] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.910185 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5069e564364a3b82e55d410fc03cec" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:55.910781 23273 raft_consensus.cc:3037] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.915832 23396 leader_election.cc:288] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.916469 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5069e564364a3b82e55d410fc03cec" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
W20240502 14:05:55.917892 23162 leader_election.cc:341] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.921854 23273 raft_consensus.cc:2454] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:55.923552 23159 leader_election.cc:302] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:55.924260 23396 raft_consensus.cc:2781] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:55.924767 23391 tablet_bootstrap.cc:492] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.924702 23396 raft_consensus.cc:686] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:05:55.925622 23396 consensus_queue.cc:237] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.927601 23391 ts_tablet_manager.cc:1367] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.072s user 0.046s sys 0.017s
I20240502 14:05:55.927503 23418 raft_consensus.cc:482] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.928084 23418 raft_consensus.cc:504] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.910871 23389 tablet_bootstrap.cc:654] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.929908 23418 leader_election.cc:288] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.930516 23395 raft_consensus.cc:482] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.930142 23391 raft_consensus.cc:348] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.930977 23395 raft_consensus.cc:504] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.931231 23391 raft_consensus.cc:374] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.932091 23391 raft_consensus.cc:729] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.932713 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8d132cd4e545b29c79863b5aa25f5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.933020 23395 leader_election.cc:288] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.933329 23196 raft_consensus.cc:2454] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:55.933001 23391 consensus_queue.cc:260] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:55.934798 23388 tablet_bootstrap.cc:492] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:55.934952 23238 leader_election.cc:302] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:55.935655 23388 ts_tablet_manager.cc:1367] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.053s user 0.024s sys 0.015s
I20240502 14:05:55.936266 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.936524 23391 ts_tablet_manager.cc:1398] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:05:55.936882 23347 raft_consensus.cc:2454] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:55.937449 23391 tablet_bootstrap.cc:492] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:55.932087 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8d132cd4e545b29c79863b5aa25f5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.940289 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:55.941546 23238 leader_election.cc:341] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.942188 23238 leader_election.cc:302] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:55.942883 23416 raft_consensus.cc:2781] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.943277 23391 tablet_bootstrap.cc:654] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:55.943511 23418 raft_consensus.cc:2781] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.943869 23418 raft_consensus.cc:482] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.944207 23418 raft_consensus.cc:3037] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.945077 23455 raft_consensus.cc:482] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.943285 23416 raft_consensus.cc:482] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.954275 23416 raft_consensus.cc:3037] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.958206 23418 raft_consensus.cc:504] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:55.947984 23238 leader_election.cc:341] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:55.951036 23393 raft_consensus.cc:482] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.952131 23089 catalog_manager.cc:5484] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:55.952293 23388 raft_consensus.cc:348] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.960258 23388 raft_consensus.cc:374] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.960701 23388 raft_consensus.cc:729] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.960036 23393 raft_consensus.cc:504] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.963318 23395 raft_consensus.cc:482] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.961300 23388 consensus_queue.cc:260] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.963843 23395 raft_consensus.cc:504] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.963899 23393 leader_election.cc:288] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:55.965610 23416 raft_consensus.cc:504] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.967341 23414 raft_consensus.cc:482] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.967847 23414 raft_consensus.cc:504] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.967021 23455 raft_consensus.cc:504] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.969763 23414 leader_election.cc:288] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:55.972685 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e158eeeafc149a58edb5538c52bbbdf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.973331 23273 raft_consensus.cc:2454] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:55.975472 23310 leader_election.cc:302] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:55.976217 23393 raft_consensus.cc:2781] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.976357 23347 raft_consensus.cc:3037] T 35e4ee45e707479b874fab4903df88f6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.976598 23393 raft_consensus.cc:482] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.976991 23393 raft_consensus.cc:3037] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.980226 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee3ba4c1e8843d8b94029ece423580f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:55.980823 23273 raft_consensus.cc:2454] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:55.981709 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee3ba4c1e8843d8b94029ece423580f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:55.981890 23159 leader_election.cc:302] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:55.982297 23346 raft_consensus.cc:2454] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:55.982882 23460 raft_consensus.cc:2781] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:55.983289 23393 raft_consensus.cc:504] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.984046 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e158eeeafc149a58edb5538c52bbbdf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:55.984651 23196 raft_consensus.cc:2454] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:55.985064 23391 tablet_bootstrap.cc:492] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:55.985531 23391 ts_tablet_manager.cc:1367] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.048s user 0.022s sys 0.021s
I20240502 14:05:55.986306 23393 leader_election.cc:288] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:55.983304 23460 raft_consensus.cc:482] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:55.987257 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e158eeeafc149a58edb5538c52bbbdf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:55.987937 23196 raft_consensus.cc:3037] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.987294 23460 raft_consensus.cc:3037] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.995973 23391 raft_consensus.cc:348] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:55.996660 23391 raft_consensus.cc:374] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:55.986840 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e158eeeafc149a58edb5538c52bbbdf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:55.996935 23391 raft_consensus.cc:729] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:55.997462 23273 raft_consensus.cc:3037] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:55.997876 23391 consensus_queue.cc:260] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.000002 23391 ts_tablet_manager.cc:1398] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 14:05:56.000885 23391 tablet_bootstrap.cc:492] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.009727 23273 raft_consensus.cc:2454] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:56.011539 23395 leader_election.cc:288] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.012269 23395 raft_consensus.cc:482] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.012710 23395 raft_consensus.cc:504] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.013900 23196 raft_consensus.cc:2454] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:56.014513 23395 leader_election.cc:288] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.014953 23313 leader_election.cc:302] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.015964 23447 raft_consensus.cc:2781] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.016458 23447 raft_consensus.cc:686] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:56.020256 23455 leader_election.cc:288] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.022442 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.025492 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e158eeeafc149a58edb5538c52bbbdf" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.025585 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff6e89be70bb4088ae2fb35610efbbbc" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.026400 23193 raft_consensus.cc:2454] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.027781 23238 leader_election.cc:302] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.029570 23453 raft_consensus.cc:482] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:55.997865 23393 raft_consensus.cc:482] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.030053 23453 raft_consensus.cc:504] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } }
I20240502 14:05:56.026404 23194 raft_consensus.cc:2379] T 0e158eeeafc149a58edb5538c52bbbdf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82490d85cd3d4a1180dc3ecabdd88598 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:56.034485 23388 ts_tablet_manager.cc:1398] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.099s user 0.010s sys 0.012s
I20240502 14:05:56.036290 23388 tablet_bootstrap.cc:492] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.029985 23416 leader_election.cc:288] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.029438 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.033843 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8d132cd4e545b29c79863b5aa25f5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.039628 23418 leader_election.cc:288] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.034433 23455 raft_consensus.cc:2781] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.017241 23447 consensus_queue.cc:237] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.041555 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff6e89be70bb4088ae2fb35610efbbbc" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:05:56.044731 23238 leader_election.cc:341] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.047339 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.036744 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e158eeeafc149a58edb5538c52bbbdf" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.045331 23455 raft_consensus.cc:482] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.049739 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.050100 23455 raft_consensus.cc:3037] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.052544 23388 tablet_bootstrap.cc:654] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.038563 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8d132cd4e545b29c79863b5aa25f5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.043017 23197 raft_consensus.cc:2454] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.035970 23393 raft_consensus.cc:504] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.045814 23396 consensus_queue.cc:1035] T 35e4ee45e707479b874fab4903df88f6 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240502 14:05:56.054839 23344 raft_consensus.cc:2454] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.058255 23393 leader_election.cc:288] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.058801 23345 raft_consensus.cc:2454] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.058938 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.053819 23343 raft_consensus.cc:3037] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.060066 23197 raft_consensus.cc:2454] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.061233 23313 leader_election.cc:302] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
W20240502 14:05:56.055974 23238 leader_election.cc:341] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.056547 23453 leader_election.cc:288] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.063488 23238 leader_election.cc:302] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.042320 23460 raft_consensus.cc:504] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.066140 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b50f554edf4267b72624ec5b865c2e" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.067551 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.068243 23273 raft_consensus.cc:2454] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.067793 23346 raft_consensus.cc:2379] T 58b50f554edf4267b72624ec5b865c2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:56.062958 23196 raft_consensus.cc:3037] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.057783 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b50f554edf4267b72624ec5b865c2e" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.064571 23393 raft_consensus.cc:2781] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.072754 23238 leader_election.cc:302] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.073437 23090 catalog_manager.cc:5484] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:56.075662 23460 leader_election.cc:288] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.070117 23459 raft_consensus.cc:2781] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.071655 23393 raft_consensus.cc:482] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.078353 23393 raft_consensus.cc:3037] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.078763 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee3ba4c1e8843d8b94029ece423580f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:56.079334 23273 raft_consensus.cc:3037] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.080215 23455 raft_consensus.cc:504] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.076565 23416 raft_consensus.cc:2726] T 0e158eeeafc149a58edb5538c52bbbdf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.082293 23455 leader_election.cc:288] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.082598 23416 raft_consensus.cc:482] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.082953 23416 raft_consensus.cc:504] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.085392 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff6e89be70bb4088ae2fb35610efbbbc" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.094815 23159 leader_election.cc:302] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:56.096089 23460 raft_consensus.cc:3037] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.099398 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff6e89be70bb4088ae2fb35610efbbbc" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.099982 23345 raft_consensus.cc:3037] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.100456 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80905fa745417489dcee47a31635fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.101047 23194 raft_consensus.cc:2454] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.102061 23238 leader_election.cc:302] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.102586 23393 raft_consensus.cc:504] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.103981 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80905fa745417489dcee47a31635fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.104568 23347 raft_consensus.cc:2454] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.104758 23393 leader_election.cc:288] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.093740 23197 raft_consensus.cc:3037] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.099522 23416 leader_election.cc:288] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.106600 23418 raft_consensus.cc:2781] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.106968 23418 raft_consensus.cc:482] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.107312 23418 raft_consensus.cc:3037] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.108629 23196 raft_consensus.cc:2454] T 1b8d132cd4e545b29c79863b5aa25f5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:56.109387 23273 raft_consensus.cc:2454] T 9ee3ba4c1e8843d8b94029ece423580f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:56.111873 23343 raft_consensus.cc:2454] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:56.111725 23238 leader_election.cc:302] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:56.081952 23459 raft_consensus.cc:482] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.114521 23459 raft_consensus.cc:3037] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.114658 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:56.113323 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee3ba4c1e8843d8b94029ece423580f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.115796 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.116034 23346 raft_consensus.cc:3037] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.116436 23196 raft_consensus.cc:3037] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.114022 23391 tablet_bootstrap.cc:654] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.114970 23389 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.119971 23389 ts_tablet_manager.cc:1367] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.227s user 0.010s sys 0.030s
I20240502 14:05:56.112221 23159 leader_election.cc:302] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.115547 23416 raft_consensus.cc:2781] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.123574 23418 raft_consensus.cc:504] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.124042 23416 raft_consensus.cc:686] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:56.124807 23346 raft_consensus.cc:2454] T 9ee3ba4c1e8843d8b94029ece423580f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:56.114624 23238 leader_election.cc:302] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.126581 23455 raft_consensus.cc:2781] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.127085 23455 raft_consensus.cc:686] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:56.127658 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80905fa745417489dcee47a31635fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.128422 23194 raft_consensus.cc:3037] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.127964 23455 consensus_queue.cc:237] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.128935 23460 raft_consensus.cc:2726] T 58b50f554edf4267b72624ec5b865c2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.121729 23414 raft_consensus.cc:2781] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.130136 23414 raft_consensus.cc:686] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:05:56.131772 23414 consensus_queue.cc:237] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.123276 23389 raft_consensus.cc:348] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.121013 23197 raft_consensus.cc:2454] T ff6e89be70bb4088ae2fb35610efbbbc P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:56.124847 23416 consensus_queue.cc:237] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.141971 23238 leader_election.cc:302] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.135509 23389 raft_consensus.cc:374] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.144402 23389 raft_consensus.cc:729] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.145021 23389 consensus_queue.cc:260] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.159436 23461 raft_consensus.cc:2781] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.159927 23461 raft_consensus.cc:686] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:56.160718 23461 consensus_queue.cc:237] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.153684 23418 leader_election.cc:288] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.167716 23389 ts_tablet_manager.cc:1398] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20240502 14:05:56.169533 23389 tablet_bootstrap.cc:492] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.171890 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80905fa745417489dcee47a31635fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.172685 23344 raft_consensus.cc:3037] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.151068 23345 raft_consensus.cc:2454] T ff6e89be70bb4088ae2fb35610efbbbc P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:56.176172 23194 raft_consensus.cc:2454] T ac80905fa745417489dcee47a31635fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:56.178794 23416 consensus_queue.cc:1035] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.179076 23090 catalog_manager.cc:5484] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:56.178725 23238 leader_election.cc:302] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.176085 23089 catalog_manager.cc:5484] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:56.182112 23091 catalog_manager.cc:5484] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:56.172176 23196 raft_consensus.cc:2454] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:56.185796 23344 raft_consensus.cc:2454] T ac80905fa745417489dcee47a31635fa P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:56.185860 23313 leader_election.cc:302] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.186897 23393 raft_consensus.cc:2781] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.187137 23459 raft_consensus.cc:504] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.187471 23393 raft_consensus.cc:686] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:56.188318 23393 consensus_queue.cc:237] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.188930 23416 raft_consensus.cc:2781] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.191030 23272 raft_consensus.cc:2379] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:56.191143 23416 raft_consensus.cc:686] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:56.197477 23459 leader_election.cc:288] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.199227 23389 tablet_bootstrap.cc:654] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.203405 23388 tablet_bootstrap.cc:492] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.203819 23388 ts_tablet_manager.cc:1367] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.168s user 0.013s sys 0.009s
I20240502 14:05:56.206004 23388 raft_consensus.cc:348] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.206634 23388 raft_consensus.cc:374] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.206920 23388 raft_consensus.cc:729] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.207502 23388 consensus_queue.cc:260] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.209179 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.210043 23196 raft_consensus.cc:2379] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 82490d85cd3d4a1180dc3ecabdd88598 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:56.213824 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6c31e65498481696374e87a81890fa" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.191965 23416 consensus_queue.cc:237] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.224436 23238 leader_election.cc:302] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.225653 23459 consensus_queue.cc:1035] T 58b50f554edf4267b72624ec5b865c2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.226836 23459 raft_consensus.cc:482] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.227303 23459 raft_consensus.cc:504] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.229101 23459 leader_election.cc:288] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.229535 23459 raft_consensus.cc:2726] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.235272 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.235495 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:56.236960 23238 leader_election.cc:341] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.237378 23388 ts_tablet_manager.cc:1398] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
W20240502 14:05:56.237592 23238 leader_election.cc:341] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.238039 23238 leader_election.cc:302] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.238265 23388 tablet_bootstrap.cc:492] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.238811 23418 raft_consensus.cc:2726] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.239163 23461 consensus_queue.cc:1035] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.248411 23391 tablet_bootstrap.cc:492] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.248869 23391 ts_tablet_manager.cc:1367] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.248s user 0.012s sys 0.009s
I20240502 14:05:56.251737 23391 raft_consensus.cc:348] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.252382 23391 raft_consensus.cc:374] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.257913 23418 consensus_queue.cc:1035] T 009254b7af044911807829cd2423373e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.261231 23090 catalog_manager.cc:5484] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:56.252696 23391 raft_consensus.cc:729] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.263728 23391 consensus_queue.cc:260] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.265642 23391 ts_tablet_manager.cc:1398] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240502 14:05:56.266443 23391 tablet_bootstrap.cc:492] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.271373 23391 tablet_bootstrap.cc:654] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.275424 23388 tablet_bootstrap.cc:654] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.280990 23391 tablet_bootstrap.cc:492] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.281389 23391 ts_tablet_manager.cc:1367] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240502 14:05:56.281590 23091 catalog_manager.cc:5484] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:56.283883 23091 catalog_manager.cc:5484] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:56.285534 23091 catalog_manager.cc:5484] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:56.289208 23389 tablet_bootstrap.cc:492] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.289577 23389 ts_tablet_manager.cc:1367] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.021s
I20240502 14:05:56.290616 23388 tablet_bootstrap.cc:492] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.291004 23388 ts_tablet_manager.cc:1367] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.006s
I20240502 14:05:56.283640 23391 raft_consensus.cc:348] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.291690 23391 raft_consensus.cc:374] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.292019 23391 raft_consensus.cc:729] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.291831 23389 raft_consensus.cc:348] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.292548 23389 raft_consensus.cc:374] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.292825 23389 raft_consensus.cc:729] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.292872 23391 consensus_queue.cc:260] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.293205 23388 raft_consensus.cc:348] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.293853 23388 raft_consensus.cc:374] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.293509 23389 consensus_queue.cc:260] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.294124 23388 raft_consensus.cc:729] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.294836 23388 consensus_queue.cc:260] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.295580 23391 ts_tablet_manager.cc:1398] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20240502 14:05:56.296356 23389 ts_tablet_manager.cc:1398] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:05:56.296444 23391 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.297204 23389 tablet_bootstrap.cc:492] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.297398 23388 ts_tablet_manager.cc:1398] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:05:56.298161 23388 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.302531 23391 tablet_bootstrap.cc:654] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.302764 23388 tablet_bootstrap.cc:654] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.302546 23389 tablet_bootstrap.cc:654] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.313223 23393 consensus_queue.cc:1035] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.316033 23414 raft_consensus.cc:482] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.316474 23414 raft_consensus.cc:504] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.318267 23414 leader_election.cc:288] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.320570 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.322774 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7774595b1544e5391939b585e1f40c8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.323585 23345 raft_consensus.cc:2379] T e7774595b1544e5391939b585e1f40c8 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:56.325474 23162 leader_election.cc:302] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:56.326665 23414 raft_consensus.cc:3037] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.327689 23389 tablet_bootstrap.cc:492] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.328137 23389 ts_tablet_manager.cc:1367] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.002s
I20240502 14:05:56.329150 23388 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.329530 23388 ts_tablet_manager.cc:1367] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20240502 14:05:56.330482 23389 raft_consensus.cc:348] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.331120 23389 raft_consensus.cc:374] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.331396 23389 raft_consensus.cc:729] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.331844 23388 raft_consensus.cc:348] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.332062 23389 consensus_queue.cc:260] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.332490 23388 raft_consensus.cc:374] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.333178 23388 raft_consensus.cc:729] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.333822 23388 consensus_queue.cc:260] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.336066 23389 ts_tablet_manager.cc:1398] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:05:56.336891 23389 tablet_bootstrap.cc:492] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.338961 23391 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.340263 23391 ts_tablet_manager.cc:1367] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.014s
I20240502 14:05:56.341722 23389 tablet_bootstrap.cc:654] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.339730 23414 raft_consensus.cc:2726] T e7774595b1544e5391939b585e1f40c8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.345034 23388 ts_tablet_manager.cc:1398] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20240502 14:05:56.345817 23391 raft_consensus.cc:348] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.346539 23391 raft_consensus.cc:374] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.346794 23391 raft_consensus.cc:729] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.346897 23389 tablet_bootstrap.cc:492] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.347362 23389 ts_tablet_manager.cc:1367] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 14:05:56.347457 23391 consensus_queue.cc:260] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.349403 23391 ts_tablet_manager.cc:1398] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:05:56.349623 23389 raft_consensus.cc:348] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.350236 23389 raft_consensus.cc:374] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.350495 23389 raft_consensus.cc:729] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.351632 23388 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.350509 23391 tablet_bootstrap.cc:492] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.357221 23388 tablet_bootstrap.cc:654] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.357529 23389 consensus_queue.cc:260] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.359737 23389 ts_tablet_manager.cc:1398] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240502 14:05:56.360584 23389 tablet_bootstrap.cc:492] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.362830 23194 raft_consensus.cc:3037] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.366251 23389 tablet_bootstrap.cc:654] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.366662 23391 tablet_bootstrap.cc:654] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.375424 23448 consensus_queue.cc:1035] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:56.377421 23388 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.377866 23388 ts_tablet_manager.cc:1367] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.010s
I20240502 14:05:56.380230 23388 raft_consensus.cc:348] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.380728 23391 tablet_bootstrap.cc:492] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.380920 23388 raft_consensus.cc:374] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.381242 23391 ts_tablet_manager.cc:1367] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.001s
I20240502 14:05:56.381244 23388 raft_consensus.cc:729] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.382084 23388 consensus_queue.cc:260] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.384074 23388 ts_tablet_manager.cc:1398] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240502 14:05:56.383782 23391 raft_consensus.cc:348] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.384450 23391 raft_consensus.cc:374] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.384714 23391 raft_consensus.cc:729] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.384907 23388 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.385360 23391 consensus_queue.cc:260] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.390189 23388 tablet_bootstrap.cc:654] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.393735 23391 ts_tablet_manager.cc:1398] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 14:05:56.396339 23389 tablet_bootstrap.cc:492] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.396785 23389 ts_tablet_manager.cc:1367] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.002s
I20240502 14:05:56.399318 23389 raft_consensus.cc:348] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.399947 23389 raft_consensus.cc:374] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.400193 23389 raft_consensus.cc:729] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.400754 23389 consensus_queue.cc:260] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.401400 23388 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.401770 23388 ts_tablet_manager.cc:1367] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240502 14:05:56.403923 23388 raft_consensus.cc:348] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.404536 23388 raft_consensus.cc:374] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.404778 23388 raft_consensus.cc:729] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.405778 23389 ts_tablet_manager.cc:1398] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240502 14:05:56.405459 23388 consensus_queue.cc:260] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.406579 23389 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.407737 23391 tablet_bootstrap.cc:492] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.411686 23389 tablet_bootstrap.cc:654] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.412089 23388 ts_tablet_manager.cc:1398] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240502 14:05:56.413015 23388 tablet_bootstrap.cc:492] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.415293 23391 tablet_bootstrap.cc:654] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.420792 23388 tablet_bootstrap.cc:654] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.424950 23389 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.425387 23389 ts_tablet_manager.cc:1367] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.001s
I20240502 14:05:56.426076 23391 tablet_bootstrap.cc:492] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.426518 23391 ts_tablet_manager.cc:1367] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20240502 14:05:56.427784 23389 raft_consensus.cc:348] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.428490 23389 raft_consensus.cc:374] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.428788 23389 raft_consensus.cc:729] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.429517 23389 consensus_queue.cc:260] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.431500 23389 ts_tablet_manager.cc:1398] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:56.432363 23389 tablet_bootstrap.cc:492] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.434732 23448 raft_consensus.cc:482] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.435238 23448 raft_consensus.cc:504] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.437150 23448 leader_election.cc:288] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.437695 23389 tablet_bootstrap.cc:654] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.438977 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5069e564364a3b82e55d410fc03cec" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.439800 23272 raft_consensus.cc:2379] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:05:56.428795 23391 raft_consensus.cc:348] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.440563 23391 raft_consensus.cc:374] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.440799 23391 raft_consensus.cc:729] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.441318 23391 consensus_queue.cc:260] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.443186 23391 ts_tablet_manager.cc:1398] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20240502 14:05:56.444000 23391 tablet_bootstrap.cc:492] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.444850 23388 tablet_bootstrap.cc:492] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.443715 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5069e564364a3b82e55d410fc03cec" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.445196 23388 ts_tablet_manager.cc:1367] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.010s
I20240502 14:05:56.446144 23313 leader_election.cc:302] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.447157 23448 raft_consensus.cc:3037] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.447386 23388 raft_consensus.cc:348] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.447908 23391 tablet_bootstrap.cc:654] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.448043 23388 raft_consensus.cc:374] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.448313 23388 raft_consensus.cc:729] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.448906 23388 consensus_queue.cc:260] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.453359 23448 raft_consensus.cc:2726] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.455286 23388 ts_tablet_manager.cc:1398] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240502 14:05:56.456187 23388 tablet_bootstrap.cc:492] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.457330 23389 tablet_bootstrap.cc:492] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.457769 23389 ts_tablet_manager.cc:1367] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20240502 14:05:56.460486 23389 raft_consensus.cc:348] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.461172 23389 raft_consensus.cc:374] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.461341 23388 tablet_bootstrap.cc:654] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.463946 23389 raft_consensus.cc:729] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.464603 23389 consensus_queue.cc:260] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.466754 23389 ts_tablet_manager.cc:1398] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240502 14:05:56.470124 23391 tablet_bootstrap.cc:492] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.470407 23389 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.470515 23391 ts_tablet_manager.cc:1367] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.007s
I20240502 14:05:56.472712 23391 raft_consensus.cc:348] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.473984 23391 raft_consensus.cc:374] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.474249 23391 raft_consensus.cc:729] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.474862 23391 consensus_queue.cc:260] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.479301 23448 raft_consensus.cc:482] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.479933 23391 ts_tablet_manager.cc:1398] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240502 14:05:56.479740 23448 raft_consensus.cc:504] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.480744 23391 tablet_bootstrap.cc:492] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.481650 23389 tablet_bootstrap.cc:654] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.482172 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf9e842a284cf8bcc8d60d0dd1a22b" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.483348 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf9e842a284cf8bcc8d60d0dd1a22b" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.483956 23272 raft_consensus.cc:2454] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.485802 23310 leader_election.cc:302] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:56.486260 23391 tablet_bootstrap.cc:654] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.486668 23447 raft_consensus.cc:2781] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.487059 23447 raft_consensus.cc:482] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.487393 23447 raft_consensus.cc:3037] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:56.491855 23313 leader_election.cc:341] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.492533 23388 tablet_bootstrap.cc:492] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.492949 23388 ts_tablet_manager.cc:1367] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.008s
I20240502 14:05:56.493245 23448 leader_election.cc:288] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.493485 23391 tablet_bootstrap.cc:492] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:56.495262 23388 raft_consensus.cc:348] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.495586 23447 raft_consensus.cc:504] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.496434 23391 ts_tablet_manager.cc:1367] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20240502 14:05:56.496371 23388 raft_consensus.cc:374] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.496994 23388 raft_consensus.cc:729] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.497658 23388 consensus_queue.cc:260] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.498426 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf9e842a284cf8bcc8d60d0dd1a22b" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:05:56.499661 23313 leader_election.cc:341] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.500293 23388 ts_tablet_manager.cc:1398] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240502 14:05:56.501498 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf9e842a284cf8bcc8d60d0dd1a22b" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:56.502017 23272 raft_consensus.cc:3037] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.505985 23448 raft_consensus.cc:482] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.506611 23447 leader_election.cc:288] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.506479 23448 raft_consensus.cc:504] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.510002 23391 raft_consensus.cc:348] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.510304 23448 leader_election.cc:288] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.510537 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c72f195f8ab449e939a764c5d237d81" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.510670 23391 raft_consensus.cc:374] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 14:05:56.511839 23310 leader_election.cc:341] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.512396 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c72f195f8ab449e939a764c5d237d81" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.513022 23194 raft_consensus.cc:2454] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.515590 23388 tablet_bootstrap.cc:492] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.515627 23313 leader_election.cc:302] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:05:56.511154 23391 raft_consensus.cc:729] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.517623 23448 raft_consensus.cc:2781] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.518043 23448 raft_consensus.cc:482] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.517915 23391 consensus_queue.cc:260] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.518486 23448 raft_consensus.cc:3037] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.517073 23460 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:05:56.520952 23416 raft_consensus.cc:482] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.521425 23391 ts_tablet_manager.cc:1398] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.025s user 0.005s sys 0.002s
I20240502 14:05:56.521391 23416 raft_consensus.cc:504] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.522969 23391 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:56.523571 23416 leader_election.cc:288] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.524940 23272 raft_consensus.cc:2454] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:56.525192 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c7a15545d94e5c8931cb1c26d3f3db" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.526587 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c7a15545d94e5c8931cb1c26d3f3db" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:56.527397 23238 leader_election.cc:341] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:05:56.528067 23238 leader_election.cc:341] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.528911 23238 leader_election.cc:302] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.529423 23310 leader_election.cc:302] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:56.531688 23447 raft_consensus.cc:2781] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.532151 23447 raft_consensus.cc:686] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:56.532866 23447 consensus_queue.cc:237] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.534924 23416 raft_consensus.cc:2726] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:56.535993 23460 consensus_queue.cc:1035] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:05:56.539072 23393 consensus_queue.cc:1035] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.539914 23388 tablet_bootstrap.cc:654] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.549381 23346 raft_consensus.cc:3037] T 1b8d132cd4e545b29c79863b5aa25f5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.558727 23416 consensus_queue.cc:1035] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:56.567776 23461 consensus_queue.cc:1035] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.568518 23448 raft_consensus.cc:504] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.571043 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c72f195f8ab449e939a764c5d237d81" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.571031 23460 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:56.570952 23389 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:56.572324 23196 raft_consensus.cc:3037] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.578599 23461 raft_consensus.cc:482] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.579049 23461 raft_consensus.cc:504] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.578886 23196 raft_consensus.cc:2454] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:56.580502 23389 ts_tablet_manager.cc:1367] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.110s user 0.032s sys 0.007s
I20240502 14:05:56.582446 23461 leader_election.cc:288] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.583886 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc011e53d9184101a0e9b06c61f73c93" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.586948 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc011e53d9184101a0e9b06c61f73c93" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.587613 23448 leader_election.cc:288] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.587749 23347 raft_consensus.cc:2454] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.583869 23313 leader_election.cc:302] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.588719 23391 tablet_bootstrap.cc:654] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.589287 23448 raft_consensus.cc:2781] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.589840 23448 raft_consensus.cc:686] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:56.590850 23448 consensus_queue.cc:237] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.588596 23460 raft_consensus.cc:482] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.593200 23238 leader_election.cc:302] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:56.593345 23460 raft_consensus.cc:504] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.588734 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c72f195f8ab449e939a764c5d237d81" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
W20240502 14:05:56.595904 23310 leader_election.cc:341] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:05:56.594064 23238 leader_election.cc:341] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.598282 23460 leader_election.cc:288] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.598948 23418 consensus_queue.cc:1035] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240502 14:05:56.611639 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ff7f6ffd34055abb266d2b25e2e4d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.616111 23389 raft_consensus.cc:348] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.616809 23389 raft_consensus.cc:374] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.617074 23389 raft_consensus.cc:729] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.617686 23389 consensus_queue.cc:260] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.618558 23344 raft_consensus.cc:2454] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:56.599175 23461 raft_consensus.cc:2781] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.619422 23461 raft_consensus.cc:482] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.619763 23461 raft_consensus.cc:3037] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.623731 23389 ts_tablet_manager.cc:1398] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20240502 14:05:56.628208 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ff7f6ffd34055abb266d2b25e2e4d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.628849 23270 raft_consensus.cc:2454] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:56.629036 23162 leader_election.cc:302] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.629841 23453 raft_consensus.cc:2781] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.630206 23453 raft_consensus.cc:482] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.630537 23453 raft_consensus.cc:3037] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.637478 23395 consensus_queue.cc:1035] T ff6e89be70bb4088ae2fb35610efbbbc P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.623823 23459 raft_consensus.cc:482] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.641746 23459 raft_consensus.cc:504] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.650822 23459 leader_election.cc:288] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.659643 23393 raft_consensus.cc:482] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.660117 23393 raft_consensus.cc:504] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.661957 23393 leader_election.cc:288] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.664045 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f8be05930c456fa1f16fe18fdc7d92" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.664671 23270 raft_consensus.cc:2454] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.668738 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2ba63147fa4983ae30a46ed72e9568" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.670214 23393 raft_consensus.cc:482] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.670681 23193 raft_consensus.cc:2454] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:56.670765 23393 raft_consensus.cc:504] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.672832 23310 leader_election.cc:302] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:56.672878 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2ba63147fa4983ae30a46ed72e9568" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.673954 23447 raft_consensus.cc:2781] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.643707 23414 consensus_queue.cc:1035] T 9ee3ba4c1e8843d8b94029ece423580f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20240502 14:05:56.675477 23447 raft_consensus.cc:482] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.675849 23447 raft_consensus.cc:3037] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.679514 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f8be05930c456fa1f16fe18fdc7d92" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.680137 23193 raft_consensus.cc:2454] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.682940 23389 tablet_bootstrap.cc:492] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:56.705770 23459 consensus_queue.cc:1035] T e7774595b1544e5391939b585e1f40c8 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:56.709721 23238 leader_election.cc:302] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:56.710486 23455 consensus_queue.cc:1035] T 1b8d132cd4e545b29c79863b5aa25f5f P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.703892 23393 leader_election.cc:288] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.716235 23393 consensus_queue.cc:1035] T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:56.724025 23461 raft_consensus.cc:504] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.725006 23388 tablet_bootstrap.cc:492] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:56.715418 23395 raft_consensus.cc:2781] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.725548 23388 ts_tablet_manager.cc:1367] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.210s user 0.017s sys 0.030s
I20240502 14:05:56.710687 23453 raft_consensus.cc:504] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.724984 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:56.729035 23269 raft_consensus.cc:2454] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:56.729087 23461 leader_election.cc:288] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
W20240502 14:05:56.722616 23238 leader_election.cc:341] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:56.726819 23395 raft_consensus.cc:482] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.726958 23459 raft_consensus.cc:482] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.749554 23388 raft_consensus.cc:348] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.750399 23388 raft_consensus.cc:374] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:56.751024 23388 raft_consensus.cc:729] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:56.755067 23388 consensus_queue.cc:260] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.764654 23388 ts_tablet_manager.cc:1398] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20240502 14:05:56.742548 23395 raft_consensus.cc:3037] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.748025 23459 raft_consensus.cc:504] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.777130 23459 leader_election.cc:288] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.765578 23388 tablet_bootstrap.cc:492] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:56.780004 23416 raft_consensus.cc:482] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:56.780439 23416 raft_consensus.cc:504] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.784474 23395 raft_consensus.cc:504] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.786345 23395 leader_election.cc:288] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.747830 23310 leader_election.cc:302] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
W20240502 14:05:56.789482 23310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 53761.4us
I20240502 14:05:56.789575 23448 raft_consensus.cc:2781] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:56.790007 23448 raft_consensus.cc:482] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:56.790344 23448 raft_consensus.cc:3037] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.790948 23447 raft_consensus.cc:504] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.792810 23447 leader_election.cc:288] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.799983 23448 raft_consensus.cc:504] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:56.800778 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f8be05930c456fa1f16fe18fdc7d92" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:56.811331 23269 raft_consensus.cc:3037] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.822077 23389 tablet_bootstrap.cc:654] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:56.821791 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc011e53d9184101a0e9b06c61f73c93" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:56.824000 23343 raft_consensus.cc:3037] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.828456 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc011e53d9184101a0e9b06c61f73c93" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:56.829659 23090 catalog_manager.cc:5484] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:56.832360 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ff7f6ffd34055abb266d2b25e2e4d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:56.833622 23268 raft_consensus.cc:3037] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:56.853502 23453 leader_election.cc:288] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.837877 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:56.876899 23269 raft_consensus.cc:2454] T b1f8be05930c456fa1f16fe18fdc7d92 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:56.847309 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:56.860388 23416 leader_election.cc:288] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:56.934784 23310 leader_election.cc:302] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:56.925855 23448 leader_election.cc:288] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:56.962522 23447 raft_consensus.cc:2781] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:56.969501 23447 raft_consensus.cc:686] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
W20240502 14:05:56.970043 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:05:56.970273 23447 consensus_queue.cc:237] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.007884 23343 raft_consensus.cc:2454] T dc011e53d9184101a0e9b06c61f73c93 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:57.013804 23461 consensus_queue.cc:1035] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20240502 14:05:57.122972 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2ba63147fa4983ae30a46ed72e9568" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
W20240502 14:05:57.049412 23238 leader_election.cc:341] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:05:57.130066 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 83009.1us
I20240502 14:05:57.154706 23388 tablet_bootstrap.cc:654] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
W20240502 14:05:57.145556 23313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 471522us
I20240502 14:05:57.147732 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:57.163641 23269 raft_consensus.cc:3037] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.166184 23514 raft_consensus.cc:482] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.176120 23514 raft_consensus.cc:504] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.176854 23269 raft_consensus.cc:2454] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:57.168200 23391 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:57.178102 23391 ts_tablet_manager.cc:1367] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.655s user 0.013s sys 0.014s
W20240502 14:05:57.173687 23238 leader_election.cc:341] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.179284 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:57.175552 23268 raft_consensus.cc:2454] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:57.189939 23391 raft_consensus.cc:348] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.190428 23391 raft_consensus.cc:374] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.190613 23391 raft_consensus.cc:729] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.191453 23391 consensus_queue.cc:260] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.202039 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b7519a6f04979b29317f144bc70fe" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:57.192361 23159 leader_election.cc:302] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:57.191357 23196 raft_consensus.cc:2454] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:57.200464 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2ba63147fa4983ae30a46ed72e9568" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:57.207342 23391 ts_tablet_manager.cc:1398] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.029s user 0.004s sys 0.006s
I20240502 14:05:57.192728 23238 leader_election.cc:302] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.208235 23391 tablet_bootstrap.cc:492] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:57.203747 23514 leader_election.cc:288] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:57.209082 23194 raft_consensus.cc:3037] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.212461 23461 raft_consensus.cc:2781] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:57.210239 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b7519a6f04979b29317f144bc70fe" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:57.212273 23310 consensus_peers.cc:485] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 -> Peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Couldn't send request to peer 82490d85cd3d4a1180dc3ecabdd88598. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:57.226511 23512 raft_consensus.cc:2781] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:57.230051 23512 raft_consensus.cc:686] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
W20240502 14:05:57.222680 23238 leader_election.cc:341] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.223289 23461 raft_consensus.cc:686] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:57.227542 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:57.235524 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f8be05930c456fa1f16fe18fdc7d92" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:57.241286 23192 raft_consensus.cc:3037] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.249863 23310 leader_election.cc:302] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:57.234966 23461 consensus_queue.cc:237] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.254549 23268 raft_consensus.cc:2379] T 71b15d9239134710bfe520f70f80a2b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:57.234923 23512 consensus_queue.cc:237] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.234239 23238 leader_election.cc:302] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.259946 23389 tablet_bootstrap.cc:492] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.260641 23389 ts_tablet_manager.cc:1367] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.578s user 0.014s sys 0.008s
I20240502 14:05:57.256256 23393 raft_consensus.cc:2781] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:57.272910 23191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:57.273599 23191 raft_consensus.cc:3037] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:57.276409 23162 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.009s sys 0.020s
W20240502 14:05:57.276739 23162 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.009s sys 0.020s
W20240502 14:05:57.277019 23162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 637374us
I20240502 14:05:57.267655 23416 raft_consensus.cc:2726] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:57.271136 23393 raft_consensus.cc:686] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
W20240502 14:05:57.271806 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 50116.8us
I20240502 14:05:57.294898 23388 tablet_bootstrap.cc:492] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:57.295611 23388 ts_tablet_manager.cc:1367] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.530s user 0.031s sys 0.025s
I20240502 14:05:57.296231 23389 raft_consensus.cc:348] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.297518 23389 raft_consensus.cc:374] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.297708 23389 raft_consensus.cc:729] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.299590 23196 raft_consensus.cc:3037] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.299428 23389 consensus_queue.cc:260] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.303726 23194 raft_consensus.cc:2454] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:57.304886 23389 ts_tablet_manager.cc:1398] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.044s user 0.001s sys 0.004s
I20240502 14:05:57.305728 23389 tablet_bootstrap.cc:492] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:57.312058 23393 consensus_queue.cc:237] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.319821 23192 raft_consensus.cc:2454] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:57.316469 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ff7f6ffd34055abb266d2b25e2e4d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:57.322060 23343 raft_consensus.cc:3037] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.322713 23388 raft_consensus.cc:348] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.330608 23388 raft_consensus.cc:374] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.330914 23388 raft_consensus.cc:729] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.335026 23388 consensus_queue.cc:260] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.337402 23388 ts_tablet_manager.cc:1398] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20240502 14:05:57.338301 23388 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:57.345389 23391 tablet_bootstrap.cc:654] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.315719 23448 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
W20240502 14:05:57.334532 23238 leader_election.cc:341] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.349403 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b15d9239134710bfe520f70f80a2b6" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:57.347359 23515 consensus_queue.cc:1035] T 9c72f195f8ab449e939a764c5d237d81 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 14:05:57.352417 23447 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240502 14:05:57.409648 23389 tablet_bootstrap.cc:654] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.388772 23343 raft_consensus.cc:2454] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:57.354559 23393 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.102s
W20240502 14:05:57.419816 23529 log.cc:924] Time spent T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20240502 14:05:57.384006 23238 leader_election.cc:302] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:57.462004 23388 tablet_bootstrap.cc:654] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.456605 23525 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:57.420958 23191 raft_consensus.cc:2454] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
W20240502 14:05:57.476976 23526 log.cc:924] Time spent T 0e158eeeafc149a58edb5538c52bbbdf P 980e09c2959e408db93d72b7125ccf63: Append to log took a long time: real 0.189s user 0.004s sys 0.000s
W20240502 14:05:57.468458 23313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 139121us
W20240502 14:05:57.470681 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 86757.7us
I20240502 14:05:57.492723 23162 leader_election.cc:302] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:57.507524 23512 raft_consensus.cc:2726] T 71b15d9239134710bfe520f70f80a2b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:57.535378 23536 raft_consensus.cc:2781] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:57.535976 23536 raft_consensus.cc:686] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:57.536787 23536 consensus_queue.cc:237] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.562124 23391 tablet_bootstrap.cc:492] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:57.562635 23391 ts_tablet_manager.cc:1367] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.355s user 0.017s sys 0.007s
W20240502 14:05:57.563266 23162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 55357.7us
I20240502 14:05:57.577796 23391 raft_consensus.cc:348] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.578510 23391 raft_consensus.cc:374] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.578794 23391 raft_consensus.cc:729] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.579413 23391 consensus_queue.cc:260] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:57.569118 23238 leader_election.cc:341] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.584515 23536 raft_consensus.cc:482] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.599433 23389 tablet_bootstrap.cc:492] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.599911 23389 ts_tablet_manager.cc:1367] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.294s user 0.008s sys 0.027s
I20240502 14:05:57.602342 23389 raft_consensus.cc:348] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.603017 23389 raft_consensus.cc:374] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.603822 23389 raft_consensus.cc:729] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.604434 23389 consensus_queue.cc:260] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.605708 23391 ts_tablet_manager.cc:1398] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.035s user 0.003s sys 0.002s
I20240502 14:05:57.606623 23391 tablet_bootstrap.cc:492] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:57.610620 23389 ts_tablet_manager.cc:1398] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20240502 14:05:57.611742 23389 tablet_bootstrap.cc:492] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:57.616379 23416 consensus_queue.cc:1035] T ac80905fa745417489dcee47a31635fa P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:57.605271 23536 raft_consensus.cc:504] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.620857 23536 leader_election.cc:288] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:57.621351 23270 raft_consensus.cc:3037] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:57.626125 23238 leader_election.cc:341] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.626595 23238 leader_election.cc:302] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.631942 23388 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:57.632380 23388 ts_tablet_manager.cc:1367] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.294s user 0.018s sys 0.018s
I20240502 14:05:57.629806 23461 raft_consensus.cc:2726] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:57.636806 23388 raft_consensus.cc:348] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.630492 23525 consensus_queue.cc:1035] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 14:05:57.640211 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c72f195f8ab449e939a764c5d237d81" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:57.640836 23388 raft_consensus.cc:374] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.641141 23388 raft_consensus.cc:729] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.643815 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c72f195f8ab449e939a764c5d237d81" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:57.643332 23388 consensus_queue.cc:260] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.648058 23389 tablet_bootstrap.cc:654] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.649214 23535 raft_consensus.cc:482] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.652606 23388 ts_tablet_manager.cc:1398] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20240502 14:05:57.653270 23388 tablet_bootstrap.cc:492] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:57.661630 23238 leader_election.cc:302] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.658555 23535 raft_consensus.cc:504] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.668661 23391 tablet_bootstrap.cc:654] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.684827 23525 consensus_queue.cc:1035] T b1f8be05930c456fa1f16fe18fdc7d92 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:57.685503 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2ba63147fa4983ae30a46ed72e9568" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:57.686236 23193 raft_consensus.cc:2379] T 5e2ba63147fa4983ae30a46ed72e9568 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:57.687222 23535 leader_election.cc:288] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:57.688654 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2ba63147fa4983ae30a46ed72e9568" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:57.701916 23313 leader_election.cc:302] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.702828 23535 raft_consensus.cc:3037] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.709272 23388 tablet_bootstrap.cc:654] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.710263 23461 raft_consensus.cc:2726] T 9c72f195f8ab449e939a764c5d237d81 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:57.713347 23535 raft_consensus.cc:2726] T 5e2ba63147fa4983ae30a46ed72e9568 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:57.718819 23092 catalog_manager.cc:5484] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:57.721064 23388 tablet_bootstrap.cc:492] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:57.722882 23388 ts_tablet_manager.cc:1367] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.003s
I20240502 14:05:57.733175 23389 tablet_bootstrap.cc:492] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.733513 23389 ts_tablet_manager.cc:1367] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.122s user 0.015s sys 0.011s
I20240502 14:05:57.733541 23388 raft_consensus.cc:348] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.734241 23388 raft_consensus.cc:374] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.734499 23388 raft_consensus.cc:729] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.735091 23388 consensus_queue.cc:260] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.735502 23389 raft_consensus.cc:348] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.736171 23389 raft_consensus.cc:374] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.736434 23389 raft_consensus.cc:729] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.737386 23388 ts_tablet_manager.cc:1398] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 14:05:57.737087 23389 consensus_queue.cc:260] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.738601 23388 tablet_bootstrap.cc:492] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:57.742358 23389 ts_tablet_manager.cc:1398] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:05:57.743180 23389 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:57.743991 23388 tablet_bootstrap.cc:654] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.744666 23391 tablet_bootstrap.cc:492] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:57.745114 23391 ts_tablet_manager.cc:1367] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.139s user 0.013s sys 0.001s
I20240502 14:05:57.745998 23461 raft_consensus.cc:482] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.746512 23461 raft_consensus.cc:504] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.747850 23389 tablet_bootstrap.cc:654] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.748657 23461 leader_election.cc:288] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:57.750445 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b400e3ddeee74798b411ee7a4e285b7a" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:57.751780 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b400e3ddeee74798b411ee7a4e285b7a" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:57.752290 23391 raft_consensus.cc:348] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.753006 23391 raft_consensus.cc:374] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.753273 23391 raft_consensus.cc:729] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
W20240502 14:05:57.753531 23238 leader_election.cc:341] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:05:57.754200 23238 leader_election.cc:341] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.754587 23238 leader_election.cc:302] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.755713 23461 raft_consensus.cc:2726] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:57.776662 23389 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.777096 23389 ts_tablet_manager.cc:1367] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.011s
I20240502 14:05:57.779491 23389 raft_consensus.cc:348] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.780130 23389 raft_consensus.cc:374] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.780377 23389 raft_consensus.cc:729] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.780970 23389 consensus_queue.cc:260] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.783103 23389 ts_tablet_manager.cc:1398] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:57.783969 23389 tablet_bootstrap.cc:492] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:57.788947 23389 tablet_bootstrap.cc:654] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.799073 23391 consensus_queue.cc:260] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.805840 23461 consensus_queue.cc:1035] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:57.808943 23389 tablet_bootstrap.cc:492] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.812335 23389 ts_tablet_manager.cc:1367] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20240502 14:05:57.814100 23391 ts_tablet_manager.cc:1398] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.069s user 0.006s sys 0.000s
I20240502 14:05:57.814963 23389 raft_consensus.cc:348] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.827328 23389 raft_consensus.cc:374] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.827689 23389 raft_consensus.cc:729] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.826978 23090 catalog_manager.cc:5484] T 9c72f195f8ab449e939a764c5d237d81 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } }
I20240502 14:05:57.828593 23194 raft_consensus.cc:3037] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.828449 23389 consensus_queue.cc:260] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.829073 23090 catalog_manager.cc:5484] T b1f8be05930c456fa1f16fe18fdc7d92 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } }
I20240502 14:05:57.830617 23389 ts_tablet_manager.cc:1398] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240502 14:05:57.831568 23389 tablet_bootstrap.cc:492] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:57.823096 23391 tablet_bootstrap.cc:492] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:57.834767 23518 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:57.837613 23391 tablet_bootstrap.cc:654] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.843134 23461 consensus_queue.cc:1035] T dc011e53d9184101a0e9b06c61f73c93 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:57.844655 23389 tablet_bootstrap.cc:654] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.849093 23512 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:57.865334 23389 tablet_bootstrap.cc:492] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.865725 23389 ts_tablet_manager.cc:1367] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.001s
I20240502 14:05:57.868235 23389 raft_consensus.cc:348] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.868906 23389 raft_consensus.cc:374] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.869163 23389 raft_consensus.cc:729] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.865712 23388 tablet_bootstrap.cc:492] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:57.869904 23388 ts_tablet_manager.cc:1367] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.131s user 0.065s sys 0.041s
I20240502 14:05:57.872423 23389 consensus_queue.cc:260] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.874681 23389 ts_tablet_manager.cc:1398] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:05:57.876343 23389 tablet_bootstrap.cc:492] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:57.880573 23525 consensus_queue.cc:1035] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:57.887732 23389 tablet_bootstrap.cc:654] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:57.889087 23090 catalog_manager.cc:5484] T dc011e53d9184101a0e9b06c61f73c93 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } }
I20240502 14:05:57.890147 23388 raft_consensus.cc:348] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.890796 23388 raft_consensus.cc:374] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:57.891067 23388 raft_consensus.cc:729] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.876499 23091 catalog_manager.cc:5484] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:57.893944 23388 consensus_queue.cc:260] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.894551 23525 consensus_queue.cc:1035] T 71b15d9239134710bfe520f70f80a2b6 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:57.896036 23388 ts_tablet_manager.cc:1398] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240502 14:05:57.896891 23388 tablet_bootstrap.cc:492] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:57.899507 23525 raft_consensus.cc:482] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.906519 23525 raft_consensus.cc:504] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.909222 23525 leader_election.cc:288] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:57.912899 23535 raft_consensus.cc:482] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.913403 23535 raft_consensus.cc:504] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.915319 23535 leader_election.cc:288] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:57.915838 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630762fac4d4b6c99da94bc7b0f77cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:57.916605 23193 raft_consensus.cc:2454] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:57.917840 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630762fac4d4b6c99da94bc7b0f77cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:57.921445 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:57.922421 23269 raft_consensus.cc:2454] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:57.926234 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:57.923025 23310 leader_election.cc:341] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.933045 23310 leader_election.cc:302] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:05:57.940889 23525 raft_consensus.cc:482] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.941566 23525 raft_consensus.cc:504] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.943637 23525 leader_election.cc:288] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
W20240502 14:05:57.944255 23313 leader_election.cc:341] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.945959 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b7519a6f04979b29317f144bc70fe" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:57.947417 23092 catalog_manager.cc:5484] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:57.949012 23535 raft_consensus.cc:2781] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:57.949360 23535 raft_consensus.cc:482] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:57.949677 23535 raft_consensus.cc:3037] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.949862 23310 leader_election.cc:302] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.950616 23448 raft_consensus.cc:2781] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:57.950942 23448 raft_consensus.cc:482] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:57.951251 23448 raft_consensus.cc:3037] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.951905 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b7519a6f04979b29317f144bc70fe" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:57.952518 23269 raft_consensus.cc:2454] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:57.953490 23388 tablet_bootstrap.cc:654] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
W20240502 14:05:57.951279 23313 leader_election.cc:341] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.960018 23536 raft_consensus.cc:482] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.962222 23536 raft_consensus.cc:504] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.963014 23448 raft_consensus.cc:504] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.964063 23536 leader_election.cc:288] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:57.964864 23448 leader_election.cc:288] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:57.965332 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac6f78a24a0432cae83b3b832145059" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:57.974045 23238 leader_election.cc:341] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:57.977671 23535 raft_consensus.cc:504] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:57.971544 23310 leader_election.cc:302] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:57.995922 23391 tablet_bootstrap.cc:492] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:57.996408 23448 raft_consensus.cc:2781] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:57.997884 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:57.997506 23389 tablet_bootstrap.cc:492] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:57.998868 23389 ts_tablet_manager.cc:1367] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.124s user 0.015s sys 0.008s
I20240502 14:05:57.998786 23268 raft_consensus.cc:3037] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:57.999456 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.001495 23389 raft_consensus.cc:348] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.002161 23389 raft_consensus.cc:374] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.002421 23389 raft_consensus.cc:729] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:57.998330 23448 raft_consensus.cc:482] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.003458 23448 raft_consensus.cc:3037] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.003672 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac6f78a24a0432cae83b3b832145059" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:57.997944 23391 ts_tablet_manager.cc:1367] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.175s user 0.024s sys 0.004s
I20240502 14:05:58.006743 23391 raft_consensus.cc:348] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.013850 23391 raft_consensus.cc:374] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.014184 23391 raft_consensus.cc:729] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.007862 23268 raft_consensus.cc:2454] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.009104 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630762fac4d4b6c99da94bc7b0f77cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.004303 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630762fac4d4b6c99da94bc7b0f77cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:05:58.010175 23313 leader_election.cc:341] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.002866 23535 leader_election.cc:288] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.003013 23389 consensus_queue.cc:260] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.017264 23341 raft_consensus.cc:2454] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:58.018222 23389 ts_tablet_manager.cc:1398] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20240502 14:05:58.019140 23389 tablet_bootstrap.cc:492] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.022686 23536 consensus_queue.cc:1035] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.023386 23193 raft_consensus.cc:3037] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.023953 23391 consensus_queue.cc:260] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:58.037998 23310 leader_election.cc:341] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.038738 23310 leader_election.cc:302] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.013152 23448 raft_consensus.cc:504] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.041743 23388 tablet_bootstrap.cc:492] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.042186 23388 ts_tablet_manager.cc:1367] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.145s user 0.008s sys 0.009s
I20240502 14:05:58.042901 23238 leader_election.cc:302] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.044262 23416 raft_consensus.cc:2781] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.044615 23416 raft_consensus.cc:482] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.044942 23416 raft_consensus.cc:3037] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.045516 23448 leader_election.cc:288] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.046177 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b7519a6f04979b29317f144bc70fe" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.046766 23196 raft_consensus.cc:3037] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.047360 23525 raft_consensus.cc:2781] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.047852 23525 raft_consensus.cc:686] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.048640 23525 consensus_queue.cc:237] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.053207 23196 raft_consensus.cc:2454] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.054241 23313 leader_election.cc:302] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:58.054587 23193 raft_consensus.cc:2454] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.055058 23447 raft_consensus.cc:2781] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.055516 23447 raft_consensus.cc:686] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.055627 23313 leader_election.cc:302] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:05:58.056555 23539 raft_consensus.cc:2781] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.056288 23447 consensus_queue.cc:237] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.057184 23539 raft_consensus.cc:686] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.058068 23539 consensus_queue.cc:237] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:58.061414 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:05:58.049341 23388 raft_consensus.cc:348] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.062347 23388 raft_consensus.cc:374] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.062620 23388 raft_consensus.cc:729] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.064221 23388 consensus_queue.cc:260] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.075352 23388 ts_tablet_manager.cc:1398] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20240502 14:05:58.076200 23388 tablet_bootstrap.cc:492] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.038962 23391 ts_tablet_manager.cc:1398] T 271b7519a6f04979b29317f144bc70fe P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20240502 14:05:58.079524 23416 raft_consensus.cc:504] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.081794 23416 leader_election.cc:288] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.049216 23389 tablet_bootstrap.cc:654] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.085610 23391 tablet_bootstrap.cc:492] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.086450 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac6f78a24a0432cae83b3b832145059" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:05:58.087945 23238 leader_election.cc:341] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.089334 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b7519a6f04979b29317f144bc70fe" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.089934 23268 raft_consensus.cc:3037] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.100584 23268 raft_consensus.cc:2454] T 271b7519a6f04979b29317f144bc70fe P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.103791 23391 tablet_bootstrap.cc:654] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.106974 23388 tablet_bootstrap.cc:654] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.110476 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac6f78a24a0432cae83b3b832145059" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:58.111085 23345 raft_consensus.cc:3037] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.114404 23534 consensus_queue.cc:1035] T 5e2ba63147fa4983ae30a46ed72e9568 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.120786 23345 raft_consensus.cc:2454] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:58.122776 23089 catalog_manager.cc:5484] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:58.124536 23089 catalog_manager.cc:5484] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:58.129395 23238 leader_election.cc:302] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.128321 23089 catalog_manager.cc:5484] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:58.130503 23416 raft_consensus.cc:2781] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.130957 23416 raft_consensus.cc:686] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:58.131775 23416 consensus_queue.cc:237] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.149888 23389 tablet_bootstrap.cc:492] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.150250 23389 ts_tablet_manager.cc:1367] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.131s user 0.012s sys 0.013s
I20240502 14:05:58.152457 23389 raft_consensus.cc:348] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.153124 23389 raft_consensus.cc:374] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.153236 23388 tablet_bootstrap.cc:492] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.153422 23389 raft_consensus.cc:729] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.153723 23388 ts_tablet_manager.cc:1367] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.013s
I20240502 14:05:58.154085 23389 consensus_queue.cc:260] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.155994 23388 raft_consensus.cc:348] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.156670 23388 raft_consensus.cc:374] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.156924 23388 raft_consensus.cc:729] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.157608 23388 consensus_queue.cc:260] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.156562 23389 ts_tablet_manager.cc:1398] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:58.159396 23389 tablet_bootstrap.cc:492] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.159934 23388 ts_tablet_manager.cc:1398] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:58.161207 23388 tablet_bootstrap.cc:492] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.165885 23416 raft_consensus.cc:482] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.166409 23416 raft_consensus.cc:504] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.166627 23090 catalog_manager.cc:5484] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:58.169512 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630762fac4d4b6c99da94bc7b0f77cf" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.170361 23193 raft_consensus.cc:2379] T 2630762fac4d4b6c99da94bc7b0f77cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82490d85cd3d4a1180dc3ecabdd88598 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:58.170749 23389 tablet_bootstrap.cc:654] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.172116 23416 leader_election.cc:288] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.175324 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2630762fac4d4b6c99da94bc7b0f77cf" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.177052 23238 leader_election.cc:302] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.178718 23388 tablet_bootstrap.cc:654] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.184576 23534 raft_consensus.cc:3037] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.187526 23447 raft_consensus.cc:482] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.188059 23447 raft_consensus.cc:504] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.188932 23389 tablet_bootstrap.cc:492] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.189273 23389 ts_tablet_manager.cc:1367] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.002s
I20240502 14:05:58.190021 23447 leader_election.cc:288] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.190984 23389 raft_consensus.cc:348] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.191573 23389 raft_consensus.cc:374] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.191834 23389 raft_consensus.cc:729] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.192569 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c7a15545d94e5c8931cb1c26d3f3db" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.192406 23389 consensus_queue.cc:260] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.193213 23268 raft_consensus.cc:2454] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:58.194114 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c7a15545d94e5c8931cb1c26d3f3db" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.194458 23389 ts_tablet_manager.cc:1398] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:05:58.195128 23310 leader_election.cc:302] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
W20240502 14:05:58.196089 23313 leader_election.cc:341] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.197896 23525 raft_consensus.cc:2781] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.198259 23525 raft_consensus.cc:482] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.198594 23525 raft_consensus.cc:3037] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.199697 23534 raft_consensus.cc:2726] T 2630762fac4d4b6c99da94bc7b0f77cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:58.200984 23391 tablet_bootstrap.cc:492] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.201306 23391 ts_tablet_manager.cc:1367] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.011s
I20240502 14:05:58.202919 23391 raft_consensus.cc:348] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.203433 23391 raft_consensus.cc:374] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.203704 23391 raft_consensus.cc:729] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.204284 23391 consensus_queue.cc:260] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.206239 23391 ts_tablet_manager.cc:1398] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:05:58.207397 23389 tablet_bootstrap.cc:492] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.208119 23447 raft_consensus.cc:482] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.209321 23388 tablet_bootstrap.cc:492] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.209817 23388 ts_tablet_manager.cc:1367] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.013s
I20240502 14:05:58.210691 23525 raft_consensus.cc:504] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.209656 23447 raft_consensus.cc:504] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.212674 23525 leader_election.cc:288] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.212462 23388 raft_consensus.cc:348] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.213214 23388 raft_consensus.cc:374] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.213536 23388 raft_consensus.cc:729] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.213809 23391 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.213614 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c7a15545d94e5c8931cb1c26d3f3db" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.214299 23388 consensus_queue.cc:260] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.215023 23268 raft_consensus.cc:3037] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.216742 23388 ts_tablet_manager.cc:1398] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:05:58.217644 23388 tablet_bootstrap.cc:492] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.219678 23391 tablet_bootstrap.cc:654] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.221920 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c7a15545d94e5c8931cb1c26d3f3db" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.222491 23193 raft_consensus.cc:3037] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.236724 23391 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.236730 23388 tablet_bootstrap.cc:654] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.236702 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53a02221c4845399218ffb0554d4044" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.238955 23391 ts_tablet_manager.cc:1367] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.008s
W20240502 14:05:58.239295 23313 leader_election.cc:341] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.239495 23268 raft_consensus.cc:2454] T e5c7a15545d94e5c8931cb1c26d3f3db P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.240594 23310 leader_election.cc:302] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:58.241073 23193 raft_consensus.cc:2454] T e5c7a15545d94e5c8931cb1c26d3f3db P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.241221 23447 leader_election.cc:288] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.241901 23447 raft_consensus.cc:2781] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.242198 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53a02221c4845399218ffb0554d4044" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.242835 23268 raft_consensus.cc:2454] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:58.242398 23447 raft_consensus.cc:686] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.243781 23310 leader_election.cc:302] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.243712 23391 raft_consensus.cc:348] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.245070 23525 raft_consensus.cc:2781] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.245478 23525 raft_consensus.cc:482] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.245822 23525 raft_consensus.cc:3037] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.246295 23388 tablet_bootstrap.cc:492] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.246755 23388 ts_tablet_manager.cc:1367] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20240502 14:05:58.249135 23388 raft_consensus.cc:348] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.249774 23388 raft_consensus.cc:374] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.250025 23388 raft_consensus.cc:729] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.243863 23447 consensus_queue.cc:237] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.252243 23391 raft_consensus.cc:374] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.251865 23388 consensus_queue.cc:260] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.252524 23391 raft_consensus.cc:729] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.252785 23389 tablet_bootstrap.cc:654] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.253471 23391 consensus_queue.cc:260] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.254380 23388 ts_tablet_manager.cc:1398] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:05:58.251848 23525 raft_consensus.cc:504] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.255551 23388 tablet_bootstrap.cc:492] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.260644 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53a02221c4845399218ffb0554d4044" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.261322 23388 tablet_bootstrap.cc:654] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
W20240502 14:05:58.261775 23313 leader_election.cc:341] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.262740 23525 leader_election.cc:288] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.264659 23391 ts_tablet_manager.cc:1398] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20240502 14:05:58.265399 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53a02221c4845399218ffb0554d4044" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.268705 23391 tablet_bootstrap.cc:492] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.265925 23268 raft_consensus.cc:3037] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.274571 23391 tablet_bootstrap.cc:654] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.278492 23389 tablet_bootstrap.cc:492] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.279100 23389 ts_tablet_manager.cc:1367] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.001s
I20240502 14:05:58.278674 23268 raft_consensus.cc:2454] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.281692 23389 raft_consensus.cc:348] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.282438 23389 raft_consensus.cc:374] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.282706 23389 raft_consensus.cc:729] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.283362 23389 consensus_queue.cc:260] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.284397 23310 leader_election.cc:302] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.285508 23389 ts_tablet_manager.cc:1398] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:58.285624 23525 raft_consensus.cc:2781] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.286115 23525 raft_consensus.cc:686] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.286074 23089 catalog_manager.cc:5484] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:58.286973 23525 consensus_queue.cc:237] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.289625 23389 tablet_bootstrap.cc:492] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.298035 23389 tablet_bootstrap.cc:654] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.303078 23391 tablet_bootstrap.cc:492] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.303606 23391 ts_tablet_manager.cc:1367] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.020s
I20240502 14:05:58.305969 23391 raft_consensus.cc:348] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.306538 23389 tablet_bootstrap.cc:492] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.306658 23391 raft_consensus.cc:374] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.307019 23389 ts_tablet_manager.cc:1367] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240502 14:05:58.306974 23391 raft_consensus.cc:729] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.307792 23391 consensus_queue.cc:260] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.309458 23389 raft_consensus.cc:348] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.310087 23389 raft_consensus.cc:374] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.310335 23389 raft_consensus.cc:729] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.310906 23389 consensus_queue.cc:260] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.313119 23389 ts_tablet_manager.cc:1398] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:05:58.313939 23389 tablet_bootstrap.cc:492] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.315899 23388 tablet_bootstrap.cc:492] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.317888 23391 ts_tablet_manager.cc:1398] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:05:58.318014 23388 ts_tablet_manager.cc:1367] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.012s
I20240502 14:05:58.318706 23389 tablet_bootstrap.cc:654] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.322063 23388 raft_consensus.cc:348] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.325856 23388 raft_consensus.cc:374] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.326179 23388 raft_consensus.cc:729] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.326613 23391 tablet_bootstrap.cc:492] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.326865 23388 consensus_queue.cc:260] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.329452 23389 tablet_bootstrap.cc:492] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.329890 23389 ts_tablet_manager.cc:1367] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240502 14:05:58.332511 23391 tablet_bootstrap.cc:654] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.333791 23388 ts_tablet_manager.cc:1398] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 14:05:58.334700 23388 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.330821 23089 catalog_manager.cc:5484] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:58.343161 23388 tablet_bootstrap.cc:654] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.345734 23389 raft_consensus.cc:348] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.346395 23389 raft_consensus.cc:374] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.346652 23389 raft_consensus.cc:729] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.347402 23389 consensus_queue.cc:260] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.349059 23389 ts_tablet_manager.cc:1398] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240502 14:05:58.349867 23389 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.352911 23391 tablet_bootstrap.cc:492] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.353343 23391 ts_tablet_manager.cc:1367] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.003s
I20240502 14:05:58.354632 23388 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.354739 23389 tablet_bootstrap.cc:654] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.355154 23388 ts_tablet_manager.cc:1367] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20240502 14:05:58.355953 23391 raft_consensus.cc:348] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.356604 23391 raft_consensus.cc:374] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.356860 23391 raft_consensus.cc:729] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.357430 23391 consensus_queue.cc:260] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.357889 23388 raft_consensus.cc:348] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.358778 23388 raft_consensus.cc:374] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.359391 23534 raft_consensus.cc:482] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.359843 23534 raft_consensus.cc:504] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.361647 23534 leader_election.cc:288] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.363094 23391 ts_tablet_manager.cc:1398] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 14:05:58.363756 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb09543bbb470883b667e0303657b6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.364002 23391 tablet_bootstrap.cc:492] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.364384 23345 raft_consensus.cc:2454] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:58.366178 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb09543bbb470883b667e0303657b6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.367537 23238 leader_election.cc:302] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:58.367749 23461 raft_consensus.cc:482] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.368243 23461 raft_consensus.cc:504] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.368849 23416 raft_consensus.cc:2781] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.369221 23416 raft_consensus.cc:482] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.369462 23416 raft_consensus.cc:3037] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.370111 23461 leader_election.cc:288] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.372246 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1848b778274b788853fcb99a639b88" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:58.373095 23238 leader_election.cc:341] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:05:58.373988 23238 leader_election.cc:341] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.359084 23388 raft_consensus.cc:729] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.375712 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1848b778274b788853fcb99a639b88" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.376061 23388 consensus_queue.cc:260] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:58.377828 23238 leader_election.cc:341] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.378283 23238 leader_election.cc:302] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.378723 23388 ts_tablet_manager.cc:1398] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20240502 14:05:58.378962 23534 raft_consensus.cc:2726] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:58.380136 23388 tablet_bootstrap.cc:492] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.380196 23416 raft_consensus.cc:504] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.381484 23391 tablet_bootstrap.cc:654] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.383256 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb09543bbb470883b667e0303657b6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.385478 23461 raft_consensus.cc:482] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.386070 23388 tablet_bootstrap.cc:654] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.385916 23461 raft_consensus.cc:504] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.386953 23391 tablet_bootstrap.cc:492] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.387388 23391 ts_tablet_manager.cc:1367] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20240502 14:05:58.387728 23461 leader_election.cc:288] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
W20240502 14:05:58.389292 23238 leader_election.cc:341] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.390264 23391 raft_consensus.cc:348] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.390913 23391 raft_consensus.cc:374] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.391294 23391 raft_consensus.cc:729] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.391976 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf069882cc4e469cd068f6da62ecbb" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.392930 23341 raft_consensus.cc:2454] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:58.395725 23238 leader_election.cc:302] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:58.390884 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf069882cc4e469cd068f6da62ecbb" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.391889 23391 consensus_queue.cc:260] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.397397 23389 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.397859 23389 ts_tablet_manager.cc:1367] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.010s
I20240502 14:05:58.397893 23534 raft_consensus.cc:2781] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.398324 23391 ts_tablet_manager.cc:1398] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:05:58.398336 23534 raft_consensus.cc:482] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.398907 23534 raft_consensus.cc:3037] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.400038 23391 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.400653 23389 raft_consensus.cc:348] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.401311 23389 raft_consensus.cc:374] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.401559 23389 raft_consensus.cc:729] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.402202 23389 consensus_queue.cc:260] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.404438 23389 ts_tablet_manager.cc:1398] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
W20240502 14:05:58.405294 23238 leader_election.cc:341] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.405727 23391 tablet_bootstrap.cc:654] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.410558 23534 raft_consensus.cc:504] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.411592 23416 leader_election.cc:288] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.414286 23534 leader_election.cc:288] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.421842 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf069882cc4e469cd068f6da62ecbb" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.422240 23388 tablet_bootstrap.cc:492] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.422690 23388 ts_tablet_manager.cc:1367] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.003s
W20240502 14:05:58.423394 23238 leader_election.cc:341] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.425209 23388 raft_consensus.cc:348] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.425704 23388 raft_consensus.cc:374] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.425873 23388 raft_consensus.cc:729] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.426290 23388 consensus_queue.cc:260] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.429168 23389 tablet_bootstrap.cc:492] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.429647 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb09543bbb470883b667e0303657b6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:58.430256 23345 raft_consensus.cc:3037] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.431349 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf069882cc4e469cd068f6da62ecbb" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:58.431955 23341 raft_consensus.cc:3037] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.436153 23388 ts_tablet_manager.cc:1398] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:05:58.436496 23389 tablet_bootstrap.cc:654] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.439416 23511 raft_consensus.cc:482] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.439855 23511 raft_consensus.cc:504] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.441493 23388 tablet_bootstrap.cc:492] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.441969 23511 leader_election.cc:288] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.443358 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8b4f12c6d843efbcc7359b62a52cf5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.443984 23347 raft_consensus.cc:2454] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:58.445188 23162 leader_election.cc:302] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:58.445511 23391 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.446102 23391 ts_tablet_manager.cc:1367] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.022s
I20240502 14:05:58.447737 23388 tablet_bootstrap.cc:654] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.448509 23391 raft_consensus.cc:348] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.449218 23391 raft_consensus.cc:374] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.449542 23391 raft_consensus.cc:729] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.449857 23511 raft_consensus.cc:2781] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.450057 23391 consensus_queue.cc:260] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.449364 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8b4f12c6d843efbcc7359b62a52cf5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.454345 23391 ts_tablet_manager.cc:1398] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
W20240502 14:05:58.454723 23159 leader_election.cc:341] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.455061 23391 tablet_bootstrap.cc:492] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.456351 23416 raft_consensus.cc:482] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.456300 23345 raft_consensus.cc:2454] T 77cb09543bbb470883b667e0303657b6 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:58.456884 23416 raft_consensus.cc:504] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.451323 23511 raft_consensus.cc:482] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.458178 23238 leader_election.cc:302] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.458343 23511 raft_consensus.cc:3037] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.459154 23416 leader_election.cc:288] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.459909 23416 raft_consensus.cc:2781] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.460064 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a156a5f03e4c8c97f11fd328ad5018" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.461004 23416 raft_consensus.cc:686] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:58.461805 23416 consensus_queue.cc:237] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.463685 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a156a5f03e4c8c97f11fd328ad5018" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.464283 23347 raft_consensus.cc:2454] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
W20240502 14:05:58.465659 23238 leader_election.cc:341] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.466398 23238 leader_election.cc:302] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.459085 23341 raft_consensus.cc:2454] T 06cf069882cc4e469cd068f6da62ecbb P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:58.467893 23391 tablet_bootstrap.cc:654] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.467674 23525 consensus_queue.cc:1035] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.468518 23389 tablet_bootstrap.cc:492] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.468955 23389 ts_tablet_manager.cc:1367] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.007s
I20240502 14:05:58.471721 23534 raft_consensus.cc:2781] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.472116 23534 raft_consensus.cc:482] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.472427 23534 raft_consensus.cc:3037] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.476400 23238 leader_election.cc:302] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.478982 23525 consensus_queue.cc:1035] T 2630762fac4d4b6c99da94bc7b0f77cf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.482112 23461 raft_consensus.cc:2781] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.482574 23461 raft_consensus.cc:686] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:58.483350 23461 consensus_queue.cc:237] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.490926 23388 tablet_bootstrap.cc:492] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.491482 23388 ts_tablet_manager.cc:1367] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.004s
I20240502 14:05:58.480410 23534 raft_consensus.cc:504] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.481125 23389 raft_consensus.cc:348] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.492123 23389 raft_consensus.cc:374] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.492391 23389 raft_consensus.cc:729] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.492996 23389 consensus_queue.cc:260] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.494038 23534 leader_election.cc:288] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.494606 23388 raft_consensus.cc:348] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.495333 23388 raft_consensus.cc:374] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.496836 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a156a5f03e4c8c97f11fd328ad5018" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.497085 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a156a5f03e4c8c97f11fd328ad5018" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:58.497272 23511 raft_consensus.cc:504] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.497604 23341 raft_consensus.cc:3037] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:05:58.498150 23238 leader_election.cc:341] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.499591 23447 raft_consensus.cc:482] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.499737 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8b4f12c6d843efbcc7359b62a52cf5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.500044 23447 raft_consensus.cc:504] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:58.501274 23159 leader_election.cc:341] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.501850 23447 leader_election.cc:288] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.503433 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1848b778274b788853fcb99a639b88" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:58.504724 23313 leader_election.cc:341] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.505790 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1848b778274b788853fcb99a639b88" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.506418 23269 raft_consensus.cc:2454] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:58.507652 23310 leader_election.cc:302] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.509069 23525 raft_consensus.cc:2781] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.509477 23525 raft_consensus.cc:482] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.509887 23525 raft_consensus.cc:3037] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.513527 23511 leader_election.cc:288] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.514832 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8b4f12c6d843efbcc7359b62a52cf5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:58.515491 23345 raft_consensus.cc:3037] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.504462 23341 raft_consensus.cc:2454] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:58.495622 23388 raft_consensus.cc:729] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.495108 23389 ts_tablet_manager.cc:1398] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240502 14:05:58.518957 23238 leader_election.cc:302] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.519754 23389 tablet_bootstrap.cc:492] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.520465 23461 raft_consensus.cc:2781] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.520993 23461 raft_consensus.cc:686] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:58.525424 23461 consensus_queue.cc:237] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.526670 23089 catalog_manager.cc:5484] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:58.528733 23447 raft_consensus.cc:482] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.529233 23447 raft_consensus.cc:504] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.529803 23389 tablet_bootstrap.cc:654] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.531440 23447 leader_election.cc:288] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.532477 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b400e3ddeee74798b411ee7a4e285b7a" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.524180 23388 consensus_queue.cc:260] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.533226 23193 raft_consensus.cc:2454] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:58.535099 23313 leader_election.cc:302] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:58.535531 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b400e3ddeee74798b411ee7a4e285b7a" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.536005 23447 raft_consensus.cc:2781] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.536151 23269 raft_consensus.cc:2454] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:05:58.536535 23447 raft_consensus.cc:482] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.536898 23447 raft_consensus.cc:3037] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.541548 23388 ts_tablet_manager.cc:1398] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20240502 14:05:58.547327 23388 tablet_bootstrap.cc:492] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.551584 23605 raft_consensus.cc:482] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.552141 23605 raft_consensus.cc:504] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.555039 23525 raft_consensus.cc:504] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.557355 23345 raft_consensus.cc:2454] T cc8b4f12c6d843efbcc7359b62a52cf5 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:58.558884 23525 leader_election.cc:288] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.570065 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1848b778274b788853fcb99a639b88" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.570593 23162 leader_election.cc:302] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:05:58.570730 23268 raft_consensus.cc:3037] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.554948 23605 leader_election.cc:288] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.572641 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1848b778274b788853fcb99a639b88" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:05:58.583483 23313 leader_election.cc:341] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.583736 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac6f78a24a0432cae83b3b832145059" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.587664 23605 raft_consensus.cc:482] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.572558 23607 raft_consensus.cc:2781] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.588124 23605 raft_consensus.cc:504] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.594312 23090 catalog_manager.cc:5484] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:58.596115 23090 catalog_manager.cc:5484] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:58.589484 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac6f78a24a0432cae83b3b832145059" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.602488 23607 raft_consensus.cc:686] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:05:58.603381 23607 consensus_queue.cc:237] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.573307 23447 raft_consensus.cc:504] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.608023 23341 raft_consensus.cc:2379] T eac6f78a24a0432cae83b3b832145059 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:58.608206 23416 consensus_queue.cc:1035] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:58.618863 23539 raft_consensus.cc:482] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.621302 23539 raft_consensus.cc:504] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.622063 23268 raft_consensus.cc:2454] T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.623028 23539 leader_election.cc:288] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.624310 23605 leader_election.cc:288] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.626062 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.628171 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e5a9e6ff8641eead910a9416773d5f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.626447 23310 leader_election.cc:302] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.630061 23539 raft_consensus.cc:2781] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.630601 23539 raft_consensus.cc:686] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.631382 23539 consensus_queue.cc:237] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.633072 23162 leader_election.cc:302] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:58.634320 23605 raft_consensus.cc:3037] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.640637 23391 tablet_bootstrap.cc:492] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.641052 23391 ts_tablet_manager.cc:1367] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.186s user 0.051s sys 0.076s
I20240502 14:05:58.646981 23447 leader_election.cc:288] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:58.609582 23511 raft_consensus.cc:482] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 14:05:58.630156 23310 leader_election.cc:341] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.684984 23511 raft_consensus.cc:504] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:58.677438 23310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 47319.9us
I20240502 14:05:58.737463 23605 raft_consensus.cc:2726] T eac6f78a24a0432cae83b3b832145059 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:58.757010 23388 tablet_bootstrap.cc:654] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.756825 23391 raft_consensus.cc:348] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.757581 23391 raft_consensus.cc:374] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.757874 23391 raft_consensus.cc:729] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.758500 23391 consensus_queue.cc:260] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.763219 23391 ts_tablet_manager.cc:1398] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.122s user 0.008s sys 0.000s
I20240502 14:05:58.769623 23391 tablet_bootstrap.cc:492] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:58.771876 23196 raft_consensus.cc:3037] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.773037 23511 leader_election.cc:288] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.776038 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5382d476949269a01d25f2e60c2d8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.776635 23268 raft_consensus.cc:2379] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:58.778285 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b400e3ddeee74798b411ee7a4e285b7a" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:58.778833 23268 raft_consensus.cc:3037] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.781899 23090 catalog_manager.cc:5484] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:58.786625 23461 consensus_queue.cc:1035] T eac6f78a24a0432cae83b3b832145059 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.761937 23448 consensus_queue.cc:1035] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20240502 14:05:58.794023 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e5a9e6ff8641eead910a9416773d5f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.798352 23416 raft_consensus.cc:482] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.797736 23092 catalog_manager.cc:5484] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:58.798836 23416 raft_consensus.cc:504] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.800716 23416 leader_election.cc:288] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.802624 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90f20f57cf34057a5e3301a044ad3ca" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:05:58.806172 23238 leader_election.cc:341] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.809427 23389 tablet_bootstrap.cc:492] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:58.809823 23389 ts_tablet_manager.cc:1367] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.290s user 0.012s sys 0.004s
I20240502 14:05:58.810858 23268 raft_consensus.cc:2454] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.812040 23389 raft_consensus.cc:348] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.812530 23389 raft_consensus.cc:374] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.812743 23389 raft_consensus.cc:729] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.813207 23389 consensus_queue.cc:260] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.814939 23389 ts_tablet_manager.cc:1398] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:05:58.815838 23389 tablet_bootstrap.cc:492] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:58.817154 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b400e3ddeee74798b411ee7a4e285b7a" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.817736 23196 raft_consensus.cc:3037] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.837230 23159 leader_election.cc:302] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:58.838224 23389 tablet_bootstrap.cc:654] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.838852 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53a02221c4845399218ffb0554d4044" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:58.842653 23605 raft_consensus.cc:2726] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:05:58.842941 23313 leader_election.cc:341] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.846076 23194 raft_consensus.cc:3037] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.848603 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90f20f57cf34057a5e3301a044ad3ca" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.851879 23268 raft_consensus.cc:2379] T a53a02221c4845399218ffb0554d4044 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:58.858042 23310 leader_election.cc:302] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:58.858162 23313 leader_election.cc:302] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.875388 23196 raft_consensus.cc:2454] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:05:58.882226 23621 raft_consensus.cc:2726] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:58.881523 23620 raft_consensus.cc:2781] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:58.891949 23388 tablet_bootstrap.cc:492] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:58.892349 23388 ts_tablet_manager.cc:1367] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.345s user 0.009s sys 0.027s
I20240502 14:05:58.898573 23622 consensus_queue.cc:1035] T 271b7519a6f04979b29317f144bc70fe P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:58.902611 23391 tablet_bootstrap.cc:654] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:58.893151 23620 raft_consensus.cc:686] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:05:58.904629 23620 consensus_queue.cc:237] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.897176 23461 raft_consensus.cc:482] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.907721 23461 raft_consensus.cc:504] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.907852 23622 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.909682 23461 leader_election.cc:288] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
W20240502 14:05:58.911185 23238 leader_election.cc:341] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.912194 23238 leader_election.cc:302] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:58.913220 23461 raft_consensus.cc:2726] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:58.912609 23621 consensus_queue.cc:1035] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:58.918802 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5b811e321d4679bb9a64f5965c6a7d" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.919518 23196 raft_consensus.cc:2454] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:58.920866 23388 raft_consensus.cc:348] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.922858 23388 raft_consensus.cc:374] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:58.923108 23388 raft_consensus.cc:729] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:58.924624 23536 raft_consensus.cc:482] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:58.931586 23536 raft_consensus.cc:504] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.924922 23238 leader_election.cc:302] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:58.934154 23461 raft_consensus.cc:2781] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:58.934525 23461 raft_consensus.cc:482] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:58.934871 23461 raft_consensus.cc:3037] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.934607 23535 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:58.935122 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:58.935995 23535 consensus_queue.cc:1035] T e5c7a15545d94e5c8931cb1c26d3f3db P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:05:58.936455 23238 leader_election.cc:341] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:58.964807 23461 raft_consensus.cc:504] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.966940 23461 leader_election.cc:288] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.979030 23536 leader_election.cc:288] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:58.979291 23388 consensus_queue.cc:260] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:58.980070 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53a02221c4845399218ffb0554d4044" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.980859 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5b811e321d4679bb9a64f5965c6a7d" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:58.981438 23388 ts_tablet_manager.cc:1398] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.089s user 0.011s sys 0.005s
I20240502 14:05:58.982029 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5b811e321d4679bb9a64f5965c6a7d" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:58.982308 23196 raft_consensus.cc:3037] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:58.988950 23388 tablet_bootstrap.cc:492] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:58.991945 23391 tablet_bootstrap.cc:492] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:58.992513 23391 ts_tablet_manager.cc:1367] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.223s user 0.011s sys 0.002s
I20240502 14:05:59.007961 23196 raft_consensus.cc:2454] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
W20240502 14:05:59.009217 23238 leader_election.cc:341] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.012763 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5b811e321d4679bb9a64f5965c6a7d" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.016634 23391 raft_consensus.cc:348] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.017813 23391 raft_consensus.cc:374] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.018060 23391 raft_consensus.cc:729] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.010535 23238 leader_election.cc:302] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.019467 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.018528 23391 consensus_queue.cc:260] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.025120 23162 leader_election.cc:302] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:59.025887 23511 raft_consensus.cc:2726] T a53a02221c4845399218ffb0554d4044 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.026968 23534 raft_consensus.cc:2781] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:59.027482 23534 raft_consensus.cc:686] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:59.028102 23534 consensus_queue.cc:237] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:59.029827 23238 leader_election.cc:341] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.030232 23089 catalog_manager.cc:5484] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:59.032119 23391 ts_tablet_manager.cc:1398] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.039s user 0.003s sys 0.004s
W20240502 14:05:59.033128 23238 consensus_peers.cc:485] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Couldn't send request to peer cc4007c6e59b4361a155c5e8a79bbfb7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:59.039476 23238 leader_election.cc:341] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.040277 23539 consensus_queue.cc:1035] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.052572 23391 tablet_bootstrap.cc:492] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.054286 23388 tablet_bootstrap.cc:654] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.058234 23391 tablet_bootstrap.cc:654] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.058655 23238 leader_election.cc:302] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.088068 23530 raft_consensus.cc:2726] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:05:59.088171 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 48725.2us
I20240502 14:05:59.095504 23389 tablet_bootstrap.cc:492] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.096001 23389 ts_tablet_manager.cc:1367] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.280s user 0.011s sys 0.011s
I20240502 14:05:59.107381 23389 raft_consensus.cc:348] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.108464 23389 raft_consensus.cc:374] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.108706 23389 raft_consensus.cc:729] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.109181 23389 consensus_queue.cc:260] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.105671 23534 raft_consensus.cc:482] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.114508 23090 catalog_manager.cc:5484] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:59.118074 23388 tablet_bootstrap.cc:492] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.118502 23388 ts_tablet_manager.cc:1367] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.130s user 0.018s sys 0.038s
I20240502 14:05:59.120036 23389 ts_tablet_manager.cc:1398] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20240502 14:05:59.120656 23389 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.132066 23530 consensus_queue.cc:1035] T 77cb09543bbb470883b667e0303657b6 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.133448 23511 raft_consensus.cc:482] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.134758 23511 raft_consensus.cc:504] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.114980 23534 raft_consensus.cc:504] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:59.138381 23238 consensus_peers.cc:485] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Couldn't send request to peer cc4007c6e59b4361a155c5e8a79bbfb7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:59.173214 23313 consensus_peers.cc:485] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 -> Peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Couldn't send request to peer cc4007c6e59b4361a155c5e8a79bbfb7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:59.176404 23534 leader_election.cc:288] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
W20240502 14:05:59.184846 23159 consensus_peers.cc:485] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Couldn't send request to peer 82490d85cd3d4a1180dc3ecabdd88598. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:59.188688 23189 raft_consensus.cc:3037] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.191727 23511 leader_election.cc:288] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.194411 23389 tablet_bootstrap.cc:654] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.195406 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a156a5f03e4c8c97f11fd328ad5018" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.207142 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a156a5f03e4c8c97f11fd328ad5018" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.207937 23341 raft_consensus.cc:2379] T e6a156a5f03e4c8c97f11fd328ad5018 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:59.208837 23162 leader_election.cc:302] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:59.211301 23391 tablet_bootstrap.cc:492] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.211745 23391 ts_tablet_manager.cc:1367] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.159s user 0.013s sys 0.006s
W20240502 14:05:59.212190 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 71639.5us
I20240502 14:05:59.206732 23388 raft_consensus.cc:348] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.217020 23388 raft_consensus.cc:374] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.217293 23388 raft_consensus.cc:729] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.217824 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a6ce78fbc0428199f226fd9120217f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.218353 23388 consensus_queue.cc:260] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:59.220484 23238 leader_election.cc:341] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.223109 23388 ts_tablet_manager.cc:1398] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.104s user 0.004s sys 0.003s
I20240502 14:05:59.224110 23388 tablet_bootstrap.cc:492] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.226398 23391 raft_consensus.cc:348] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.226500 23615 consensus_queue.cc:1035] T cc8b4f12c6d843efbcc7359b62a52cf5 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.227111 23391 raft_consensus.cc:374] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.227429 23391 raft_consensus.cc:729] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.228045 23391 consensus_queue.cc:260] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.236861 23391 ts_tablet_manager.cc:1398] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20240502 14:05:59.240176 23391 tablet_bootstrap.cc:492] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.268038 23391 tablet_bootstrap.cc:654] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.269901 23530 consensus_queue.cc:1035] T 06cf069882cc4e469cd068f6da62ecbb P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.278190 23535 consensus_queue.cc:1035] T a53a02221c4845399218ffb0554d4044 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:05:59.278573 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:05:59.284585 23388 tablet_bootstrap.cc:654] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.293303 23389 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.293776 23389 ts_tablet_manager.cc:1367] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.173s user 0.028s sys 0.013s
I20240502 14:05:59.295311 23511 raft_consensus.cc:2726] T e6a156a5f03e4c8c97f11fd328ad5018 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.296430 23389 raft_consensus.cc:348] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.297113 23389 raft_consensus.cc:374] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.297407 23389 raft_consensus.cc:729] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.296864 23534 consensus_queue.cc:1035] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 14:05:59.301680 23389 consensus_queue.cc:260] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.303061 23534 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.303535 23534 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.305295 23534 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.308120 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.312655 23391 tablet_bootstrap.cc:492] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
W20240502 14:05:59.312887 23238 leader_election.cc:341] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.313073 23391 ts_tablet_manager.cc:1367] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.011s
I20240502 14:05:59.314429 23389 ts_tablet_manager.cc:1398] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240502 14:05:59.315351 23391 raft_consensus.cc:348] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.315752 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a6ce78fbc0428199f226fd9120217f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.316068 23391 raft_consensus.cc:374] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.316373 23391 raft_consensus.cc:729] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.317170 23391 consensus_queue.cc:260] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.315284 23389 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.320281 23391 ts_tablet_manager.cc:1398] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 14:05:59.321091 23391 tablet_bootstrap.cc:492] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.322993 23448 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.327572 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.323733 23448 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:59.331012 23238 leader_election.cc:341] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.331483 23238 leader_election.cc:302] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.331655 23447 consensus_queue.cc:1035] T fb1848b778274b788853fcb99a639b88 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.332803 23534 consensus_queue.cc:1035] T e6a156a5f03e4c8c97f11fd328ad5018 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.333957 23343 raft_consensus.cc:2454] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:59.334197 23448 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:59.334614 23536 raft_consensus.cc:2726] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.339661 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.344427 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.345014 23273 raft_consensus.cc:2454] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
W20240502 14:05:59.348335 23313 leader_election.cc:341] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.349814 23238 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.350601 23310 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.355461 23530 raft_consensus.cc:2781] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.355832 23530 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.356173 23530 raft_consensus.cc:3037] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.355319 23447 raft_consensus.cc:2781] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.359238 23620 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.358948 23447 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.361898 23447 raft_consensus.cc:3037] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.367705 23391 tablet_bootstrap.cc:654] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.369115 23389 tablet_bootstrap.cc:654] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.371431 23447 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.374388 23447 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:59.375133 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:05:59.376425 23313 leader_election.cc:341] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.384454 23530 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.387434 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:59.387523 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:59.388264 23269 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
W20240502 14:05:59.388897 23238 leader_election.cc:341] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.390794 23388 tablet_bootstrap.cc:492] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.391288 23388 ts_tablet_manager.cc:1367] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.167s user 0.011s sys 0.000s
I20240502 14:05:59.391990 23310 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.393102 23447 raft_consensus.cc:2726] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.399525 23388 raft_consensus.cc:348] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.400262 23388 raft_consensus.cc:374] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.400518 23388 raft_consensus.cc:729] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.401110 23388 consensus_queue.cc:260] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.405004 23448 raft_consensus.cc:482] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.405441 23448 raft_consensus.cc:504] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.407452 23448 leader_election.cc:288] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:59.408762 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5b811e321d4679bb9a64f5965c6a7d" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.409036 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5b811e321d4679bb9a64f5965c6a7d" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.409857 23189 raft_consensus.cc:2379] T 6b5b811e321d4679bb9a64f5965c6a7d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:59.411911 23310 leader_election.cc:302] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.412586 23448 raft_consensus.cc:3037] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.402946 23388 ts_tablet_manager.cc:1398] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:05:59.421753 23388 tablet_bootstrap.cc:492] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.424059 23448 raft_consensus.cc:2726] T 6b5b811e321d4679bb9a64f5965c6a7d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.432734 23530 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.433884 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.434818 23343 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 82490d85cd3d4a1180dc3ecabdd88598 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:59.435840 23238 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.436158 23535 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:59.436914 23530 raft_consensus.cc:2726] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.455829 23389 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.456243 23389 ts_tablet_manager.cc:1367] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.001s
I20240502 14:05:59.460409 23389 raft_consensus.cc:348] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.462150 23389 raft_consensus.cc:374] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.462393 23389 raft_consensus.cc:729] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.462930 23389 consensus_queue.cc:260] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.465556 23416 consensus_queue.cc:1035] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:59.472306 23391 tablet_bootstrap.cc:492] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.472767 23391 ts_tablet_manager.cc:1367] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.152s user 0.013s sys 0.039s
I20240502 14:05:59.476249 23389 ts_tablet_manager.cc:1398] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20240502 14:05:59.477133 23389 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.477056 23391 raft_consensus.cc:348] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.477834 23391 raft_consensus.cc:374] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.478096 23391 raft_consensus.cc:729] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.478675 23391 consensus_queue.cc:260] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.497555 23388 tablet_bootstrap.cc:654] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.498971 23615 raft_consensus.cc:482] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.503556 23391 ts_tablet_manager.cc:1398] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20240502 14:05:59.499492 23615 raft_consensus.cc:504] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.504782 23391 tablet_bootstrap.cc:492] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.506475 23615 leader_election.cc:288] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.511497 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf069882cc4e469cd068f6da62ecbb" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.516604 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf069882cc4e469cd068f6da62ecbb" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.519254 23159 leader_election.cc:302] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:59.520190 23615 raft_consensus.cc:2726] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.522078 23389 tablet_bootstrap.cc:654] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
W20240502 14:05:59.522511 23651 log.cc:924] Time spent T fb1848b778274b788853fcb99a639b88 P 82490d85cd3d4a1180dc3ecabdd88598: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20240502 14:05:59.535490 23190 raft_consensus.cc:3037] T 77cb09543bbb470883b667e0303657b6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.540069 23391 tablet_bootstrap.cc:654] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.562768 23658 consensus_queue.cc:1035] T 6b5b811e321d4679bb9a64f5965c6a7d P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:59.565054 23388 tablet_bootstrap.cc:492] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.565507 23388 ts_tablet_manager.cc:1367] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.144s user 0.020s sys 0.004s
I20240502 14:05:59.567855 23388 raft_consensus.cc:348] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.568511 23388 raft_consensus.cc:374] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.568778 23388 raft_consensus.cc:729] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.571485 23388 consensus_queue.cc:260] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.573890 23388 ts_tablet_manager.cc:1398] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240502 14:05:59.577246 23388 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.582473 23388 tablet_bootstrap.cc:654] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.583217 23389 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.583650 23389 ts_tablet_manager.cc:1367] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.107s user 0.012s sys 0.002s
I20240502 14:05:59.585975 23389 raft_consensus.cc:348] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.586591 23190 raft_consensus.cc:3037] T 06cf069882cc4e469cd068f6da62ecbb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.586647 23389 raft_consensus.cc:374] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.591537 23389 raft_consensus.cc:729] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.592227 23389 consensus_queue.cc:260] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.594377 23389 ts_tablet_manager.cc:1398] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240502 14:05:59.601760 23388 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.602257 23389 tablet_bootstrap.cc:492] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.602290 23388 ts_tablet_manager.cc:1367] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20240502 14:05:59.601738 23391 tablet_bootstrap.cc:492] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.603899 23391 ts_tablet_manager.cc:1367] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.009s
I20240502 14:05:59.604894 23388 raft_consensus.cc:348] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.605654 23388 raft_consensus.cc:374] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.605909 23388 raft_consensus.cc:729] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.606479 23388 consensus_queue.cc:260] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.606961 23391 raft_consensus.cc:348] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.607812 23391 raft_consensus.cc:374] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.608107 23391 raft_consensus.cc:729] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.608774 23391 consensus_queue.cc:260] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.611624 23391 ts_tablet_manager.cc:1398] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 14:05:59.611922 23389 tablet_bootstrap.cc:654] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.612530 23391 tablet_bootstrap.cc:492] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.612905 23388 ts_tablet_manager.cc:1398] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20240502 14:05:59.614270 23388 tablet_bootstrap.cc:492] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.617945 23391 tablet_bootstrap.cc:654] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.619412 23388 tablet_bootstrap.cc:654] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.628175 23539 raft_consensus.cc:482] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.628672 23539 raft_consensus.cc:504] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.629348 23391 tablet_bootstrap.cc:492] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.629859 23391 ts_tablet_manager.cc:1367] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20240502 14:05:59.630599 23539 leader_election.cc:288] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:05:59.632516 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b17aa98c3f04a58a5613d8b4af43f2e" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.632673 23391 raft_consensus.cc:348] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.634526 23391 raft_consensus.cc:374] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.634935 23391 raft_consensus.cc:729] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.635224 23388 tablet_bootstrap.cc:492] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
W20240502 14:05:59.635365 23310 leader_election.cc:341] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.634418 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b17aa98c3f04a58a5613d8b4af43f2e" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.635660 23391 consensus_queue.cc:260] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:05:59.636870 23313 leader_election.cc:341] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.637297 23313 leader_election.cc:302] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:05:59.638362 23391 ts_tablet_manager.cc:1398] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:05:59.639292 23391 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.639691 23539 raft_consensus.cc:2726] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.644049 23388 ts_tablet_manager.cc:1367] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.008s
I20240502 14:05:59.644237 23391 tablet_bootstrap.cc:654] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.646412 23388 raft_consensus.cc:348] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.647078 23388 raft_consensus.cc:374] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.647382 23388 raft_consensus.cc:729] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.648000 23388 consensus_queue.cc:260] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.650599 23391 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.651001 23391 ts_tablet_manager.cc:1367] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 14:05:59.652210 23388 ts_tablet_manager.cc:1398] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:05:59.654222 23391 raft_consensus.cc:348] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.654875 23391 raft_consensus.cc:374] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.655114 23391 raft_consensus.cc:729] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.655673 23391 consensus_queue.cc:260] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.657800 23391 ts_tablet_manager.cc:1398] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:05:59.658218 23268 raft_consensus.cc:3037] T cc8b4f12c6d843efbcc7359b62a52cf5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.658591 23391 tablet_bootstrap.cc:492] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.667490 23391 tablet_bootstrap.cc:654] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.668407 23388 tablet_bootstrap.cc:492] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.669469 23389 tablet_bootstrap.cc:492] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.670063 23389 ts_tablet_manager.cc:1367] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.020s
I20240502 14:05:59.677603 23388 tablet_bootstrap.cc:654] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.677243 23389 raft_consensus.cc:348] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.680820 23391 tablet_bootstrap.cc:492] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.681574 23389 raft_consensus.cc:374] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.681730 23391 ts_tablet_manager.cc:1367] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.007s
I20240502 14:05:59.681893 23389 raft_consensus.cc:729] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.682562 23389 consensus_queue.cc:260] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.683305 23190 raft_consensus.cc:3037] T fb1848b778274b788853fcb99a639b88 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.683753 23391 raft_consensus.cc:348] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.684404 23391 raft_consensus.cc:374] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.684587 23389 ts_tablet_manager.cc:1398] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:05:59.684641 23391 raft_consensus.cc:729] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.685389 23391 consensus_queue.cc:260] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.687340 23391 ts_tablet_manager.cc:1398] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240502 14:05:59.688151 23391 tablet_bootstrap.cc:492] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.689265 23389 tablet_bootstrap.cc:492] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.693331 23391 tablet_bootstrap.cc:654] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.704558 23388 tablet_bootstrap.cc:492] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.705004 23388 ts_tablet_manager.cc:1367] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.003s
I20240502 14:05:59.707746 23388 raft_consensus.cc:348] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.708431 23388 raft_consensus.cc:374] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.708701 23388 raft_consensus.cc:729] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.709338 23388 consensus_queue.cc:260] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.711139 23388 ts_tablet_manager.cc:1398] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 14:05:59.712065 23388 tablet_bootstrap.cc:492] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.712604 23615 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.713094 23615 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.715809 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.716596 23268 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:05:59.722779 23389 tablet_bootstrap.cc:654] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.724663 23388 tablet_bootstrap.cc:654] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.726680 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.727561 23347 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:05:59.729348 23162 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:59.730129 23643 raft_consensus.cc:3037] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.732592 23615 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.739554 23643 raft_consensus.cc:2726] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:05:59.745532 23389 tablet_bootstrap.cc:492] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.745978 23389 ts_tablet_manager.cc:1367] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.004s
I20240502 14:05:59.749406 23391 tablet_bootstrap.cc:492] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.749881 23391 ts_tablet_manager.cc:1367] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.062s user 0.032s sys 0.019s
I20240502 14:05:59.749385 23389 raft_consensus.cc:348] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.750775 23389 raft_consensus.cc:374] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.751066 23389 raft_consensus.cc:729] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.751670 23389 consensus_queue.cc:260] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.752208 23391 raft_consensus.cc:348] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.753959 23391 raft_consensus.cc:374] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.754192 23391 raft_consensus.cc:729] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.754638 23388 tablet_bootstrap.cc:492] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.755054 23388 ts_tablet_manager.cc:1367] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.001s
I20240502 14:05:59.754797 23391 consensus_queue.cc:260] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.756580 23389 ts_tablet_manager.cc:1398] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240502 14:05:59.757095 23391 ts_tablet_manager.cc:1398] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:05:59.757527 23389 tablet_bootstrap.cc:492] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.758936 23391 tablet_bootstrap.cc:492] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.760326 23536 raft_consensus.cc:482] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.760813 23536 raft_consensus.cc:504] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.761633 23388 raft_consensus.cc:348] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.762415 23388 raft_consensus.cc:374] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.762677 23388 raft_consensus.cc:729] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.762850 23536 leader_election.cc:288] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.763406 23388 consensus_queue.cc:260] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.764935 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69461a11cb2240929c388c86126ad78c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.765483 23388 ts_tablet_manager.cc:1398] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:05:59.765528 23190 raft_consensus.cc:2454] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:59.766242 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69461a11cb2240929c388c86126ad78c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.766773 23238 leader_election.cc:302] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.767320 23388 tablet_bootstrap.cc:492] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
W20240502 14:05:59.767597 23238 leader_election.cc:341] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.768054 23536 raft_consensus.cc:2781] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.771811 23536 raft_consensus.cc:482] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.770166 23391 tablet_bootstrap.cc:654] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.772212 23536 raft_consensus.cc:3037] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.772828 23389 tablet_bootstrap.cc:654] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.775882 23388 tablet_bootstrap.cc:654] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.779757 23536 raft_consensus.cc:504] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.784052 23391 tablet_bootstrap.cc:492] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.784583 23391 ts_tablet_manager.cc:1367] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.001s
I20240502 14:05:59.787261 23391 raft_consensus.cc:348] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.787751 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69461a11cb2240929c388c86126ad78c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:59.788074 23391 raft_consensus.cc:374] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.788506 23391 raft_consensus.cc:729] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.789263 23391 consensus_queue.cc:260] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.791687 23643 raft_consensus.cc:482] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.791869 23391 ts_tablet_manager.cc:1398] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:05:59.792133 23643 raft_consensus.cc:504] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.793148 23391 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.794237 23643 leader_election.cc:288] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.795213 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90f20f57cf34057a5e3301a044ad3ca" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.796144 23190 raft_consensus.cc:3037] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.797379 23347 raft_consensus.cc:2454] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:59.798503 23162 leader_election.cc:302] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.796612 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90f20f57cf34057a5e3301a044ad3ca" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.799573 23268 raft_consensus.cc:2454] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:59.801187 23643 raft_consensus.cc:2781] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.801542 23643 raft_consensus.cc:482] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.801888 23643 raft_consensus.cc:3037] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.802572 23391 tablet_bootstrap.cc:654] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.816341 23190 raft_consensus.cc:2454] T 69461a11cb2240929c388c86126ad78c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:59.817489 23536 leader_election.cc:288] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.818434 23643 raft_consensus.cc:504] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.820353 23643 leader_election.cc:288] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.821413 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90f20f57cf34057a5e3301a044ad3ca" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.822029 23347 raft_consensus.cc:3037] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.826826 23388 tablet_bootstrap.cc:492] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.827322 23238 leader_election.cc:302] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.827548 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69461a11cb2240929c388c86126ad78c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.828119 23658 raft_consensus.cc:2781] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:59.828567 23658 raft_consensus.cc:686] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:59.829347 23658 consensus_queue.cc:237] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.829666 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90f20f57cf34057a5e3301a044ad3ca" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:59.830284 23269 raft_consensus.cc:3037] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.830408 23391 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.830956 23391 ts_tablet_manager.cc:1367] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.007s
W20240502 14:05:59.831176 23238 leader_election.cc:341] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.827324 23388 ts_tablet_manager.cc:1367] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.060s user 0.021s sys 0.005s
I20240502 14:05:59.834829 23391 raft_consensus.cc:348] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.835561 23391 raft_consensus.cc:374] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.835862 23391 raft_consensus.cc:729] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.834654 23388 raft_consensus.cc:348] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.836516 23269 raft_consensus.cc:2454] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:59.836833 23388 raft_consensus.cc:374] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.837035 23643 raft_consensus.cc:482] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.836658 23391 consensus_queue.cc:260] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.837494 23388 raft_consensus.cc:729] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.837529 23643 raft_consensus.cc:504] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.838316 23388 consensus_queue.cc:260] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.840073 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b2b38a668947049fc57d84a48e6bdb" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.837707 23347 raft_consensus.cc:2454] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:59.840709 23268 raft_consensus.cc:2454] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:59.841473 23162 leader_election.cc:302] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.841771 23159 leader_election.cc:302] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.842046 23388 ts_tablet_manager.cc:1398] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:05:59.842325 23615 raft_consensus.cc:2781] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:59.842543 23605 raft_consensus.cc:2781] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.842893 23605 raft_consensus.cc:482] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.843328 23391 ts_tablet_manager.cc:1398] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240502 14:05:59.843364 23605 raft_consensus.cc:3037] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.844476 23391 tablet_bootstrap.cc:492] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.850423 23391 tablet_bootstrap.cc:654] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.851688 23643 leader_election.cc:288] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.850719 23605 raft_consensus.cc:504] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.854633 23605 leader_election.cc:288] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.854969 23389 tablet_bootstrap.cc:492] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.855454 23389 ts_tablet_manager.cc:1367] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.098s user 0.018s sys 0.021s
I20240502 14:05:59.858131 23389 raft_consensus.cc:348] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.858767 23389 raft_consensus.cc:374] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.860666 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b2b38a668947049fc57d84a48e6bdb" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:59.862716 23268 raft_consensus.cc:3037] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.866267 23536 raft_consensus.cc:482] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.869684 23643 raft_consensus.cc:482] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.870218 23643 raft_consensus.cc:504] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.861454 23389 raft_consensus.cc:729] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.871546 23389 consensus_queue.cc:260] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.872671 23643 leader_election.cc:288] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.873802 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a6ce78fbc0428199f226fd9120217f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:05:59.874645 23389 ts_tablet_manager.cc:1398] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20240502 14:05:59.874449 23269 raft_consensus.cc:2454] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:59.866762 23536 raft_consensus.cc:504] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.875895 23159 leader_election.cc:302] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.876636 23536 leader_election.cc:288] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.876814 23605 raft_consensus.cc:2781] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.842907 23388 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.877903 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c123d564235427795f0271d9ae4213c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.878715 23389 tablet_bootstrap.cc:492] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.869473 23090 catalog_manager.cc:5484] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:59.879313 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c123d564235427795f0271d9ae4213c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.879958 23341 raft_consensus.cc:2454] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:59.878576 23190 raft_consensus.cc:2454] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:59.869252 23391 tablet_bootstrap.cc:492] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.881688 23238 leader_election.cc:302] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:05:59.881971 23391 ts_tablet_manager.cc:1367] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.006s
I20240502 14:05:59.882843 23536 raft_consensus.cc:2781] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.883221 23536 raft_consensus.cc:482] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.883628 23536 raft_consensus.cc:3037] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.884256 23389 tablet_bootstrap.cc:654] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.885313 23391 raft_consensus.cc:348] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.887714 23391 raft_consensus.cc:374] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.888350 23391 raft_consensus.cc:729] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.889578 23391 consensus_queue.cc:260] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.892114 23391 ts_tablet_manager.cc:1398] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:05:59.892938 23391 tablet_bootstrap.cc:492] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.890319 23536 raft_consensus.cc:504] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.895486 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c123d564235427795f0271d9ae4213c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:59.896029 23190 raft_consensus.cc:3037] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.896602 23388 tablet_bootstrap.cc:654] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.898491 23605 raft_consensus.cc:482] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.898909 23605 raft_consensus.cc:3037] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.899688 23391 tablet_bootstrap.cc:654] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.905392 23190 raft_consensus.cc:2454] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:59.906550 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b2b38a668947049fc57d84a48e6bdb" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.907145 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a6ce78fbc0428199f226fd9120217f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.907855 23345 raft_consensus.cc:2454] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:05:59.908499 23238 leader_election.cc:302] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
W20240502 14:05:59.909370 23162 leader_election.cc:341] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.909327 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b2b38a668947049fc57d84a48e6bdb" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.910430 23388 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:05:59.910864 23388 ts_tablet_manager.cc:1367] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.001s
I20240502 14:05:59.911046 23658 raft_consensus.cc:2781] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:59.911500 23391 tablet_bootstrap.cc:492] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:05:59.911964 23391 ts_tablet_manager.cc:1367] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.001s
W20240502 14:05:59.912091 23162 leader_election.cc:341] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:05:59.912683 23268 raft_consensus.cc:2454] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:59.912865 23658 raft_consensus.cc:686] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:05:59.913368 23388 raft_consensus.cc:348] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.913868 23159 leader_election.cc:302] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:05:59.914811 23391 raft_consensus.cc:348] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.915354 23658 consensus_queue.cc:237] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.916297 23643 raft_consensus.cc:2781] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:59.917740 23536 leader_election.cc:288] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.918658 23643 raft_consensus.cc:686] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:05:59.927654 23643 consensus_queue.cc:237] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.917541 23389 tablet_bootstrap.cc:492] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:05:59.917927 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c123d564235427795f0271d9ae4213c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.931257 23389 ts_tablet_manager.cc:1367] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.000s
I20240502 14:05:59.920302 23615 raft_consensus.cc:686] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:05:59.917308 23605 raft_consensus.cc:504] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.932909 23605 leader_election.cc:288] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.918387 23388 raft_consensus.cc:374] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.933424 23388 raft_consensus.cc:729] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.917896 23391 raft_consensus.cc:374] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.933889 23389 raft_consensus.cc:348] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.934031 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a6ce78fbc0428199f226fd9120217f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:05:59.934566 23389 raft_consensus.cc:374] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:59.931442 23347 raft_consensus.cc:3037] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.934980 23389 raft_consensus.cc:729] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.934285 23391 raft_consensus.cc:729] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:05:59.935663 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a6ce78fbc0428199f226fd9120217f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:05:59.934769 23268 raft_consensus.cc:3037] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.938649 23536 raft_consensus.cc:482] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:59.939100 23536 raft_consensus.cc:504] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.940930 23536 leader_election.cc:288] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:05:59.940649 23615 consensus_queue.cc:237] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.941620 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc8a96cb68549b2b10d5d759fb49b86" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:05:59.936651 23391 consensus_queue.cc:260] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.943346 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc8a96cb68549b2b10d5d759fb49b86" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:05:59.944278 23391 ts_tablet_manager.cc:1398] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20240502 14:05:59.945127 23391 tablet_bootstrap.cc:492] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:05:59.934047 23388 consensus_queue.cc:260] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.947609 23388 ts_tablet_manager.cc:1398] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20240502 14:05:59.948424 23388 tablet_bootstrap.cc:492] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:05:59.948681 23190 raft_consensus.cc:2454] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:05:59.950837 23238 leader_election.cc:302] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.951395 23345 raft_consensus.cc:3037] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.951673 23536 raft_consensus.cc:2781] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:59.952077 23536 raft_consensus.cc:482] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:59.952409 23536 raft_consensus.cc:3037] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:59.953469 23347 raft_consensus.cc:2454] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:05:59.962019 23391 tablet_bootstrap.cc:654] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.962162 23536 raft_consensus.cc:504] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.965798 23388 tablet_bootstrap.cc:654] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.968130 23345 raft_consensus.cc:2454] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:59.969484 23268 raft_consensus.cc:2454] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:05:59.970393 23162 leader_election.cc:302] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:05:59.972553 23389 consensus_queue.cc:260] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:05:59.979398 23389 ts_tablet_manager.cc:1398] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20240502 14:05:59.982324 23389 tablet_bootstrap.cc:492] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:05:59.978458 23643 raft_consensus.cc:2781] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.017263 23389 tablet_bootstrap.cc:654] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:59.981794 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc8a96cb68549b2b10d5d759fb49b86" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:05:59.979784 23342 raft_consensus.cc:2454] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.019915 23536 leader_election.cc:288] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.020120 23643 raft_consensus.cc:686] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:00.021585 23190 raft_consensus.cc:3037] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.028050 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc8a96cb68549b2b10d5d759fb49b86" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:00.030952 23641 raft_consensus.cc:482] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.033596 23641 raft_consensus.cc:504] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.035418 23641 leader_election.cc:288] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.037534 23682 raft_consensus.cc:482] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.038045 23090 catalog_manager.cc:5484] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.037953 23682 raft_consensus.cc:504] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.039721 23090 catalog_manager.cc:5484] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.043005 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e5a9e6ff8641eead910a9416773d5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.043237 23682 leader_election.cc:288] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.043601 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e5a9e6ff8641eead910a9416773d5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.044198 23193 raft_consensus.cc:2454] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.045111 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.045694 23343 raft_consensus.cc:2454] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.046052 23643 consensus_queue.cc:237] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.047080 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:06:00.049605 23159 leader_election.cc:341] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:00.055359 23162 leader_election.cc:341] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.055791 23162 leader_election.cc:302] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:00.056906 23605 raft_consensus.cc:2726] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.057814 23388 tablet_bootstrap.cc:492] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.059617 23388 ts_tablet_manager.cc:1367] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.026s
I20240502 14:06:00.061545 23190 raft_consensus.cc:2454] T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:00.063681 23391 tablet_bootstrap.cc:492] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.064113 23391 ts_tablet_manager.cc:1367] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.119s user 0.006s sys 0.019s
I20240502 14:06:00.053753 23238 leader_election.cc:302] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:06:00.065160 23530 raft_consensus.cc:2781] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.065547 23530 raft_consensus.cc:482] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.065052 23092 catalog_manager.cc:5484] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.065897 23530 raft_consensus.cc:3037] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.066282 23388 raft_consensus.cc:348] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.066674 23391 raft_consensus.cc:348] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.067744 23391 raft_consensus.cc:374] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.068068 23391 raft_consensus.cc:729] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.068696 23391 consensus_queue.cc:260] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.070751 23391 ts_tablet_manager.cc:1398] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:06:00.072326 23391 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.072820 23238 leader_election.cc:302] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:00.071352 23388 raft_consensus.cc:374] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.073379 23530 raft_consensus.cc:504] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.074451 23388 raft_consensus.cc:729] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.075371 23658 raft_consensus.cc:2781] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.076025 23658 raft_consensus.cc:686] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:00.076915 23658 consensus_queue.cc:237] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.081034 23530 leader_election.cc:288] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.081660 23388 consensus_queue.cc:260] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.082363 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e5a9e6ff8641eead910a9416773d5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:00.082966 23193 raft_consensus.cc:3037] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.084031 23388 ts_tablet_manager.cc:1398] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20240502 14:06:00.085623 23391 tablet_bootstrap.cc:654] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.083063 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e5a9e6ff8641eead910a9416773d5f" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:00.089148 23342 raft_consensus.cc:3037] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.089278 23193 raft_consensus.cc:2454] T 01e5a9e6ff8641eead910a9416773d5f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:00.090266 23238 leader_election.cc:302] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:00.090976 23530 raft_consensus.cc:2781] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.091457 23530 raft_consensus.cc:686] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:00.092217 23530 consensus_queue.cc:237] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.101752 23389 tablet_bootstrap.cc:492] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.102203 23389 ts_tablet_manager.cc:1367] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.122s user 0.010s sys 0.003s
I20240502 14:06:00.104933 23389 raft_consensus.cc:348] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.098770 23090 catalog_manager.cc:5484] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.105522 23643 raft_consensus.cc:482] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.106055 23643 raft_consensus.cc:504] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.107872 23643 leader_election.cc:288] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.105561 23389 raft_consensus.cc:374] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.109036 23389 raft_consensus.cc:729] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.109325 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f93998b6cbe40bc9dd530d7f03e8cbf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.109642 23389 consensus_queue.cc:260] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.110204 23388 tablet_bootstrap.cc:492] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:00.115324 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f93998b6cbe40bc9dd530d7f03e8cbf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.115995 23269 raft_consensus.cc:2454] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:00.117046 23159 leader_election.cc:302] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:00.118069 23643 raft_consensus.cc:2781] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.118433 23643 raft_consensus.cc:482] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.118765 23643 raft_consensus.cc:3037] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:06:00.120211 23162 leader_election.cc:341] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.116000 23388 tablet_bootstrap.cc:654] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.128950 23643 raft_consensus.cc:504] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.130448 23389 ts_tablet_manager.cc:1398] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20240502 14:06:00.130753 23342 raft_consensus.cc:2454] T 01e5a9e6ff8641eead910a9416773d5f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:00.134868 23389 tablet_bootstrap.cc:492] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.137662 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f93998b6cbe40bc9dd530d7f03e8cbf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:00.138208 23269 raft_consensus.cc:3037] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.141219 23643 leader_election.cc:288] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.142267 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f93998b6cbe40bc9dd530d7f03e8cbf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
W20240502 14:06:00.143642 23162 leader_election.cc:341] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.144287 23089 catalog_manager.cc:5484] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.146955 23388 tablet_bootstrap.cc:492] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.147408 23388 ts_tablet_manager.cc:1367] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.002s
I20240502 14:06:00.146612 23089 catalog_manager.cc:5484] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.150077 23388 raft_consensus.cc:348] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.150732 23388 raft_consensus.cc:374] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.150995 23388 raft_consensus.cc:729] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.151937 23269 raft_consensus.cc:2454] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:00.156617 23389 tablet_bootstrap.cc:654] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.156410 23388 consensus_queue.cc:260] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.158372 23388 ts_tablet_manager.cc:1398] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240502 14:06:00.160072 23391 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.160423 23391 ts_tablet_manager.cc:1367] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.088s user 0.029s sys 0.039s
I20240502 14:06:00.160297 23159 leader_election.cc:302] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:00.161103 23643 raft_consensus.cc:2781] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.161640 23388 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:00.161572 23643 raft_consensus.cc:686] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:00.163384 23391 raft_consensus.cc:348] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.164041 23391 raft_consensus.cc:374] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.164271 23391 raft_consensus.cc:729] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.162598 23643 consensus_queue.cc:237] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.169396 23658 raft_consensus.cc:482] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.169823 23658 raft_consensus.cc:504] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.171663 23658 leader_election.cc:288] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.173252 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:06:00.174501 23238 leader_election.cc:341] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.173122 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.173074 23535 raft_consensus.cc:482] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.175154 23388 tablet_bootstrap.cc:654] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.175395 23535 raft_consensus.cc:504] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:00.175824 23238 leader_election.cc:341] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.176219 23238 leader_election.cc:302] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.177140 23658 raft_consensus.cc:2726] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.178041 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.177253 23535 leader_election.cc:288] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.178591 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.179224 23269 raft_consensus.cc:2454] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
W20240502 14:06:00.179419 23313 leader_election.cc:341] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.180330 23310 leader_election.cc:302] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.182494 23535 raft_consensus.cc:2781] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.182821 23535 raft_consensus.cc:482] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.183077 23535 raft_consensus.cc:3037] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.184520 23389 tablet_bootstrap.cc:492] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.184943 23389 ts_tablet_manager.cc:1367] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.001s
I20240502 14:06:00.164819 23391 consensus_queue.cc:260] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.187266 23389 raft_consensus.cc:348] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.187916 23391 ts_tablet_manager.cc:1398] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.027s user 0.001s sys 0.004s
I20240502 14:06:00.188854 23535 raft_consensus.cc:504] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.189492 23389 raft_consensus.cc:374] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.189821 23389 raft_consensus.cc:729] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.190501 23389 consensus_queue.cc:260] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.191134 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:06:00.192389 23313 leader_election.cc:341] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.192492 23389 ts_tablet_manager.cc:1398] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:00.188864 23391 tablet_bootstrap.cc:492] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.193203 23535 leader_election.cc:288] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.193614 23389 tablet_bootstrap.cc:492] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.197959 23391 tablet_bootstrap.cc:654] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.201591 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:00.202186 23269 raft_consensus.cc:3037] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.203466 23389 tablet_bootstrap.cc:654] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.206357 23092 catalog_manager.cc:5484] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:00.208760 23391 tablet_bootstrap.cc:492] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.209200 23391 ts_tablet_manager.cc:1367] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.007s
I20240502 14:06:00.211150 23269 raft_consensus.cc:2454] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:06:00.211534 23391 raft_consensus.cc:348] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.212203 23310 leader_election.cc:302] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.212224 23391 raft_consensus.cc:374] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.212635 23391 raft_consensus.cc:729] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.212954 23448 raft_consensus.cc:2781] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.213436 23448 raft_consensus.cc:686] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:06:00.213230 23391 consensus_queue.cc:260] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.214207 23448 consensus_queue.cc:237] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.215480 23391 ts_tablet_manager.cc:1398] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 14:06:00.216344 23391 tablet_bootstrap.cc:492] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.220708 23389 tablet_bootstrap.cc:492] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.220870 23388 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.221154 23389 ts_tablet_manager.cc:1367] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.003s
I20240502 14:06:00.221266 23388 ts_tablet_manager.cc:1367] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.002s
I20240502 14:06:00.223518 23389 raft_consensus.cc:348] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.223696 23388 raft_consensus.cc:348] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.224169 23389 raft_consensus.cc:374] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.224303 23388 raft_consensus.cc:374] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.224449 23389 raft_consensus.cc:729] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.224584 23388 raft_consensus.cc:729] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.225068 23389 consensus_queue.cc:260] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.225908 23388 consensus_queue.cc:260] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.227612 23389 ts_tablet_manager.cc:1398] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 14:06:00.228411 23389 tablet_bootstrap.cc:492] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.228405 23388 ts_tablet_manager.cc:1398] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:06:00.229858 23388 tablet_bootstrap.cc:492] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:00.234259 23391 tablet_bootstrap.cc:654] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.240649 23388 tablet_bootstrap.cc:654] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.241696 23389 tablet_bootstrap.cc:654] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.235754 23090 catalog_manager.cc:5484] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:00.245579 23388 tablet_bootstrap.cc:492] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.245960 23388 ts_tablet_manager.cc:1367] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240502 14:06:00.246591 23389 tablet_bootstrap.cc:492] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.247128 23389 ts_tablet_manager.cc:1367] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20240502 14:06:00.249831 23391 tablet_bootstrap.cc:492] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.250276 23391 ts_tablet_manager.cc:1367] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.000s
I20240502 14:06:00.250063 23389 raft_consensus.cc:348] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.250720 23389 raft_consensus.cc:374] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.250985 23389 raft_consensus.cc:729] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.251586 23389 consensus_queue.cc:260] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.253134 23389 ts_tablet_manager.cc:1398] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:06:00.252892 23391 raft_consensus.cc:348] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.253540 23391 raft_consensus.cc:374] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.253847 23391 raft_consensus.cc:729] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.254309 23389 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.254492 23391 consensus_queue.cc:260] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.256546 23391 ts_tablet_manager.cc:1398] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:00.249833 23388 raft_consensus.cc:348] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.257433 23388 raft_consensus.cc:374] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.257690 23388 raft_consensus.cc:729] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.258962 23391 tablet_bootstrap.cc:492] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.258245 23388 consensus_queue.cc:260] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.261984 23389 tablet_bootstrap.cc:654] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.263564 23536 consensus_queue.cc:1035] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:00.266345 23391 tablet_bootstrap.cc:654] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.269898 23388 ts_tablet_manager.cc:1398] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.024s user 0.005s sys 0.002s
I20240502 14:06:00.270459 23388 tablet_bootstrap.cc:492] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:00.272362 23535 raft_consensus.cc:482] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.272830 23535 raft_consensus.cc:504] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.274626 23535 leader_election.cc:288] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.275099 23388 tablet_bootstrap.cc:654] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.276243 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41b81c5154af7a4644e29408cb4cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.276907 23269 raft_consensus.cc:2454] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:06:00.278592 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41b81c5154af7a4644e29408cb4cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.281767 23389 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.282177 23389 ts_tablet_manager.cc:1367] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.004s
I20240502 14:06:00.283372 23310 leader_election.cc:302] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:06:00.284183 23448 raft_consensus.cc:482] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.284497 23389 raft_consensus.cc:348] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.284641 23448 raft_consensus.cc:504] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.285169 23389 raft_consensus.cc:374] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.285553 23389 raft_consensus.cc:729] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.286154 23389 consensus_queue.cc:260] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.287147 23539 raft_consensus.cc:2781] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.287534 23539 raft_consensus.cc:482] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.287854 23539 raft_consensus.cc:3037] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.286696 23448 leader_election.cc:288] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.293895 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69461a11cb2240929c388c86126ad78c" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.293371 23190 raft_consensus.cc:2454] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:06:00.296969 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69461a11cb2240929c388c86126ad78c" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.293610 23535 raft_consensus.cc:482] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.298215 23535 raft_consensus.cc:504] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.298621 23310 leader_election.cc:302] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.299662 23448 raft_consensus.cc:2726] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.302829 23536 raft_consensus.cc:482] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.303082 23539 raft_consensus.cc:504] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.303333 23536 raft_consensus.cc:504] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.304329 23391 tablet_bootstrap.cc:492] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.304637 23391 ts_tablet_manager.cc:1367] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.005s
I20240502 14:06:00.305016 23539 leader_election.cc:288] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.305100 23536 leader_election.cc:288] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.300087 23535 leader_election.cc:288] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.306429 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41b81c5154af7a4644e29408cb4cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:00.306893 23389 ts_tablet_manager.cc:1398] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.024s user 0.002s sys 0.005s
I20240502 14:06:00.307143 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b17aa98c3f04a58a5613d8b4af43f2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.307025 23193 raft_consensus.cc:3037] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.307564 23391 raft_consensus.cc:348] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.308012 23196 raft_consensus.cc:2454] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.308279 23391 raft_consensus.cc:374] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.308596 23391 raft_consensus.cc:729] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.309253 23389 tablet_bootstrap.cc:492] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.309253 23238 leader_election.cc:302] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:00.309218 23391 consensus_queue.cc:260] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.311003 23658 raft_consensus.cc:2781] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.311385 23391 ts_tablet_manager.cc:1398] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240502 14:06:00.311369 23658 raft_consensus.cc:482] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.312207 23658 raft_consensus.cc:3037] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.312929 23391 tablet_bootstrap.cc:492] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.314674 23389 tablet_bootstrap.cc:654] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.318087 23391 tablet_bootstrap.cc:654] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.300937 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c964342c58447b955034291dc40cb6" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.322242 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41b81c5154af7a4644e29408cb4cf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:00.322808 23268 raft_consensus.cc:3037] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:06:00.328163 23313 leader_election.cc:341] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.329044 23391 tablet_bootstrap.cc:492] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.329571 23391 ts_tablet_manager.cc:1367] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240502 14:06:00.329910 23389 tablet_bootstrap.cc:492] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.329792 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b17aa98c3f04a58a5613d8b4af43f2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.330338 23389 ts_tablet_manager.cc:1367] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.004s
I20240502 14:06:00.330415 23342 raft_consensus.cc:2454] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.332546 23391 raft_consensus.cc:348] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.333184 23391 raft_consensus.cc:374] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.332759 23389 raft_consensus.cc:348] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.333468 23391 raft_consensus.cc:729] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.333602 23389 raft_consensus.cc:374] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.334029 23389 raft_consensus.cc:729] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.334241 23391 consensus_queue.cc:260] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.334707 23389 consensus_queue.cc:260] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.336194 23193 raft_consensus.cc:2454] T 25d41b81c5154af7a4644e29408cb4cf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:06:00.337209 23391 ts_tablet_manager.cc:1398] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 14:06:00.337093 23313 leader_election.cc:302] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:00.338038 23391 tablet_bootstrap.cc:492] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.340466 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c964342c58447b955034291dc40cb6" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.341468 23388 tablet_bootstrap.cc:492] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.341883 23388 ts_tablet_manager.cc:1367] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.016s
I20240502 14:06:00.342176 23535 raft_consensus.cc:482] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.342797 23643 raft_consensus.cc:482] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.342613 23535 raft_consensus.cc:504] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.343235 23643 raft_consensus.cc:504] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.344003 23388 raft_consensus.cc:348] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.344658 23388 raft_consensus.cc:374] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.344614 23535 leader_election.cc:288] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.345085 23388 raft_consensus.cc:729] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.345214 23643 leader_election.cc:288] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.346143 23388 consensus_queue.cc:260] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.346572 23448 raft_consensus.cc:2781] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.346645 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.347173 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.347797 23273 raft_consensus.cc:2454] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:00.348348 23388 ts_tablet_manager.cc:1398] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 14:06:00.349162 23388 tablet_bootstrap.cc:492] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:00.349422 23159 leader_election.cc:302] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:00.349987 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:06:00.351123 23162 leader_election.cc:341] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.351297 23193 raft_consensus.cc:2454] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:06:00.352540 23643 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:00.354060 23391 tablet_bootstrap.cc:654] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
W20240502 14:06:00.354223 23162 consensus_peers.cc:485] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Couldn't send request to peer 980e09c2959e408db93d72b7125ccf63. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:00.355427 23643 raft_consensus.cc:2781] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.355834 23643 raft_consensus.cc:482] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.355973 23313 leader_election.cc:302] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:00.356917 23643 raft_consensus.cc:3037] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.357164 23525 raft_consensus.cc:482] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.357632 23525 raft_consensus.cc:504] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.359496 23525 leader_election.cc:288] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.359558 23388 tablet_bootstrap.cc:654] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.359647 23539 raft_consensus.cc:2781] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.360165 23539 raft_consensus.cc:482] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.360518 23539 raft_consensus.cc:3037] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.361500 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f93998b6cbe40bc9dd530d7f03e8cbf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.367794 23643 raft_consensus.cc:504] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.369329 23389 ts_tablet_manager.cc:1398] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
W20240502 14:06:00.370100 23310 leader_election.cc:341] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.371754 23310 leader_election.cc:302] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.371364 23389 tablet_bootstrap.cc:492] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.376467 23268 raft_consensus.cc:2454] T 25d41b81c5154af7a4644e29408cb4cf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:06:00.378796 23658 raft_consensus.cc:504] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.387280 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b17aa98c3f04a58a5613d8b4af43f2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:00.387848 23190 raft_consensus.cc:3037] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.391623 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.375532 23535 raft_consensus.cc:2726] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.395520 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f93998b6cbe40bc9dd530d7f03e8cbf" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.426766 23273 raft_consensus.cc:2379] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:06:00.429594 23391 tablet_bootstrap.cc:492] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:00.430060 23391 ts_tablet_manager.cc:1367] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.000s
I20240502 14:06:00.427039 23658 leader_election.cc:288] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.429340 23448 raft_consensus.cc:686] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:06:00.438788 23643 leader_election.cc:288] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.442626 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:00.452308 23391 raft_consensus.cc:348] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.453439 23391 raft_consensus.cc:374] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.453714 23391 raft_consensus.cc:729] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.462508 23388 tablet_bootstrap.cc:492] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.462966 23388 ts_tablet_manager.cc:1367] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.114s user 0.031s sys 0.035s
I20240502 14:06:00.463680 23389 tablet_bootstrap.cc:654] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.464052 23539 raft_consensus.cc:504] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:00.548178 23310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 83692.8us
I20240502 14:06:00.546738 23391 consensus_queue.cc:260] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:00.552822 23310 leader_election.cc:341] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.482381 23535 raft_consensus.cc:482] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.553192 23539 leader_election.cc:288] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
W20240502 14:06:00.486368 23162 leader_election.cc:341] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:00.554024 23162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 71251.4us
I20240502 14:06:00.553421 23535 raft_consensus.cc:504] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:00.554821 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 92825.5us
I20240502 14:06:00.523427 23605 raft_consensus.cc:482] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.556303 23535 leader_election.cc:288] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.556104 23605 raft_consensus.cc:504] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.558382 23448 consensus_queue.cc:237] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.571353 23536 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:00.573864 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b17aa98c3f04a58a5613d8b4af43f2e" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:00.574414 23342 raft_consensus.cc:3037] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.569664 23658 raft_consensus.cc:482] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.574744 23388 raft_consensus.cc:348] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.572000 23310 leader_election.cc:302] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.576364 23388 raft_consensus.cc:374] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.576663 23388 raft_consensus.cc:729] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.577239 23388 consensus_queue.cc:260] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.565222 23605 leader_election.cc:288] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.568362 23190 raft_consensus.cc:2454] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:00.595510 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:00.590098 23346 raft_consensus.cc:3037] T 9fc8a96cb68549b2b10d5d759fb49b86 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.580366 23615 raft_consensus.cc:482] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.599149 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:00.580400 23658 raft_consensus.cc:504] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.599411 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b2b38a668947049fc57d84a48e6bdb" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.580778 23535 raft_consensus.cc:3037] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.605415 23388 ts_tablet_manager.cc:1398] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.142s user 0.006s sys 0.000s
I20240502 14:06:00.608480 23388 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:00.606029 23269 raft_consensus.cc:3037] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.606254 23530 raft_consensus.cc:482] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.614603 23391 ts_tablet_manager.cc:1398] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.184s user 0.013s sys 0.015s
I20240502 14:06:00.607818 23196 raft_consensus.cc:3037] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.616112 23391 tablet_bootstrap.cc:492] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:00.604204 23615 raft_consensus.cc:504] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.619676 23658 leader_election.cc:288] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.620049 23615 leader_election.cc:288] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.613134 23536 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240502 14:06:00.623520 23525 raft_consensus.cc:482] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.623955 23525 raft_consensus.cc:504] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.625778 23525 leader_election.cc:288] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:00.616592 23530 raft_consensus.cc:504] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.632686 23530 leader_election.cc:288] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.633605 23342 raft_consensus.cc:2454] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:00.642518 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.637603 23269 raft_consensus.cc:2454] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:00.653990 23535 raft_consensus.cc:2726] T 3f93998b6cbe40bc9dd530d7f03e8cbf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.649356 23726 raft_consensus.cc:482] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.658859 23726 raft_consensus.cc:504] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.660387 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.660800 23726 leader_election.cc:288] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.660989 23193 raft_consensus.cc:2454] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.658290 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.656157 23188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c964342c58447b955034291dc40cb6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.668773 23238 leader_election.cc:302] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
W20240502 14:06:00.673450 23238 leader_election.cc:341] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.674773 23188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afca5b05bac0423ca02254a903b07eca" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:06:00.677363 23313 leader_election.cc:341] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.673110 23530 raft_consensus.cc:2781] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.681890 23461 consensus_queue.cc:1035] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:00.682209 23162 leader_election.cc:341] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.683427 23530 raft_consensus.cc:686] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:00.680927 23536 consensus_queue.cc:1035] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 14:06:00.684270 23530 consensus_queue.cc:237] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.685938 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.686916 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.689141 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:00.690219 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b2b38a668947049fc57d84a48e6bdb" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.690975 23273 raft_consensus.cc:2379] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:06:00.691401 23388 tablet_bootstrap.cc:654] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.697808 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c964342c58447b955034291dc40cb6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.698457 23342 raft_consensus.cc:2454] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:00.700896 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a22b60b6bb46189f2c845b2d299a3f" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.701611 23268 raft_consensus.cc:2379] T 69a22b60b6bb46189f2c845b2d299a3f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
W20240502 14:06:00.705446 23162 leader_election.cc:341] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.706990 23188 raft_consensus.cc:3037] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.683665 23343 raft_consensus.cc:3037] T 69461a11cb2240929c388c86126ad78c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.708765 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.719718 23238 leader_election.cc:302] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
W20240502 14:06:00.723789 23238 leader_election.cc:341] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.725617 23238 leader_election.cc:302] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.727157 23731 raft_consensus.cc:2781] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.730682 23729 raft_consensus.cc:2781] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:00.731061 23729 raft_consensus.cc:482] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.731477 23729 raft_consensus.cc:3037] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.722878 23682 consensus_queue.cc:1035] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240502 14:06:00.732746 23731 raft_consensus.cc:482] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:00.724418 23461 consensus_queue.cc:1035] T 9fc8a96cb68549b2b10d5d759fb49b86 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 14:06:00.740083 23534 raft_consensus.cc:482] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.742551 23731 raft_consensus.cc:3037] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.752321 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afca5b05bac0423ca02254a903b07eca" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.759735 23389 tablet_bootstrap.cc:492] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:00.762301 23389 ts_tablet_manager.cc:1367] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.391s user 0.011s sys 0.008s
I20240502 14:06:00.753212 23534 raft_consensus.cc:504] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.775148 23391 tablet_bootstrap.cc:654] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
W20240502 14:06:00.783324 23310 leader_election.cc:341] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.784101 23310 leader_election.cc:302] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.793368 23721 raft_consensus.cc:3037] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.795075 23389 raft_consensus.cc:348] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.795816 23389 raft_consensus.cc:374] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.796080 23389 raft_consensus.cc:729] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.796792 23389 consensus_queue.cc:260] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.800029 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:00.802597 23389 ts_tablet_manager.cc:1398] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
W20240502 14:06:00.802989 23310 leader_election.cc:341] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.803443 23310 leader_election.cc:302] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:00.805310 23389 tablet_bootstrap.cc:492] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:00.805608 23089 catalog_manager.cc:5484] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:00.811774 23730 raft_consensus.cc:482] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.811864 23722 raft_consensus.cc:2726] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.818164 23730 raft_consensus.cc:504] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.809900 23416 consensus_queue.cc:1035] T 01e5a9e6ff8641eead910a9416773d5f P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240502 14:06:00.826057 23159 leader_election.cc:302] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:00.828434 23389 tablet_bootstrap.cc:654] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.834756 23159 leader_election.cc:302] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
W20240502 14:06:00.836429 23159 leader_election.cc:341] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:00.836839 23159 leader_election.cc:302] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:00.801059 23729 raft_consensus.cc:504] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.817111 23605 consensus_queue.cc:1035] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240502 14:06:00.835928 23682 raft_consensus.cc:2781] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.825693 23196 raft_consensus.cc:2454] T cd50bda89ed043c4900bd94025639e77 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:06:00.840121 23643 raft_consensus.cc:2726] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.854930 23388 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:00.855397 23388 ts_tablet_manager.cc:1367] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.247s user 0.011s sys 0.009s
W20240502 14:06:00.855312 23732 log.cc:924] Time spent T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20240502 14:06:00.857997 23725 consensus_queue.cc:1035] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:00.846865 23534 leader_election.cc:288] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.866596 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adefff536c04350a45b88a896f35226" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:00.871912 23605 consensus_queue.cc:1035] T 3f93998b6cbe40bc9dd530d7f03e8cbf P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:00.873684 23736 log.cc:924] Time spent T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20240502 14:06:00.874118 23727 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20240502 14:06:00.878820 23313 leader_election.cc:302] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:06:00.886968 23388 raft_consensus.cc:348] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.888077 23388 raft_consensus.cc:374] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.888350 23388 raft_consensus.cc:729] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.888969 23388 consensus_queue.cc:260] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.886929 23721 raft_consensus.cc:2726] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.886314 23730 leader_election.cc:288] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.894191 23448 raft_consensus.cc:2781] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.908491 23448 raft_consensus.cc:686] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:06:00.909355 23448 consensus_queue.cc:237] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:00.913614 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:06:00.898586 23191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adefff536c04350a45b88a896f35226" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.913254 23461 consensus_queue.cc:1035] T 69461a11cb2240929c388c86126ad78c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 14:06:00.906762 23682 raft_consensus.cc:686] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:00.931041 23729 leader_election.cc:288] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:00.932215 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:00.933259 23726 raft_consensus.cc:2726] T 69a22b60b6bb46189f2c845b2d299a3f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:00.927676 23658 raft_consensus.cc:482] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:00.934244 23682 consensus_queue.cc:237] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.934998 23658 raft_consensus.cc:504] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.941921 23731 raft_consensus.cc:504] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:00.945528 23535 consensus_queue.cc:1035] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:00.949447 23268 raft_consensus.cc:2379] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc4007c6e59b4361a155c5e8a79bbfb7 in current term 1: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:06:00.949410 23658 leader_election.cc:288] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
W20240502 14:06:00.961680 23747 log.cc:924] Time spent T 9fc8a96cb68549b2b10d5d759fb49b86 P cc4007c6e59b4361a155c5e8a79bbfb7: Append to log took a long time: real 0.077s user 0.004s sys 0.000s
W20240502 14:06:00.980070 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 86506.7us
W20240502 14:06:00.980908 23238 leader_election.cc:341] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:00.982309 23313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 86534.3us
I20240502 14:06:00.985915 23721 consensus_queue.cc:1035] T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:01.015208 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afca5b05bac0423ca02254a903b07eca" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.039134 23731 leader_election.cc:288] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.045428 23721 raft_consensus.cc:482] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.045996 23721 raft_consensus.cc:504] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.053817 23391 tablet_bootstrap.cc:492] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.058081 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c964342c58447b955034291dc40cb6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:01.058676 23188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.061204 23721 leader_election.cc:288] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.063367 23391 ts_tablet_manager.cc:1367] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.448s user 0.005s sys 0.026s
I20240502 14:06:01.065397 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.067646 23159 leader_election.cc:302] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
W20240502 14:06:01.073422 23313 leader_election.cc:341] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.067099 23725 consensus_queue.cc:1035] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20240502 14:06:01.076041 23238 leader_election.cc:341] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:01.076632 23238 leader_election.cc:341] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.076972 23238 leader_election.cc:302] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.079679 23388 ts_tablet_manager.cc:1398] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.224s user 0.010s sys 0.020s
W20240502 14:06:01.079808 23238 leader_election.cc:341] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.080423 23530 raft_consensus.cc:2726] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.083307 23388 tablet_bootstrap.cc:492] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:01.088160 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
W20240502 14:06:01.083781 23238 leader_election.cc:341] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:01.096938 23310 leader_election.cc:341] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.097466 23310 leader_election.cc:302] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.101179 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
W20240502 14:06:01.114625 23238 leader_election.cc:341] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.115177 23238 leader_election.cc:302] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.129555 23745 raft_consensus.cc:3037] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:06:01.180593 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 63314.5us
I20240502 14:06:01.180980 23389 tablet_bootstrap.cc:492] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:01.181897 23389 ts_tablet_manager.cc:1367] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.377s user 0.014s sys 0.054s
I20240502 14:06:01.182468 23388 tablet_bootstrap.cc:654] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.125867 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c964342c58447b955034291dc40cb6" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.184449 23534 raft_consensus.cc:2726] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.188066 23344 raft_consensus.cc:3037] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.195137 23391 raft_consensus.cc:348] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.200261 23391 raft_consensus.cc:374] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.200996 23391 raft_consensus.cc:729] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.201640 23391 consensus_queue.cc:260] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.205714 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afca5b05bac0423ca02254a903b07eca" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.206377 23342 raft_consensus.cc:2454] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:01.217267 23391 ts_tablet_manager.cc:1398] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.154s user 0.006s sys 0.001s
I20240502 14:06:01.218243 23391 tablet_bootstrap.cc:492] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
W20240502 14:06:01.286453 23310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 189541us
I20240502 14:06:01.287004 23535 raft_consensus.cc:2726] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.293329 23238 leader_election.cc:302] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.302845 23658 raft_consensus.cc:2781] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.303237 23658 raft_consensus.cc:482] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.303570 23658 raft_consensus.cc:3037] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.312227 23389 raft_consensus.cc:348] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.312981 23389 raft_consensus.cc:374] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.313230 23389 raft_consensus.cc:729] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.331467 23389 consensus_queue.cc:260] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.333102 23755 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20240502 14:06:01.302160 23092 catalog_manager.cc:5484] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:01.292785 23745 raft_consensus.cc:2726] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.376654 23389 ts_tablet_manager.cc:1398] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.194s user 0.003s sys 0.003s
I20240502 14:06:01.377615 23389 tablet_bootstrap.cc:492] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:01.393494 23388 tablet_bootstrap.cc:492] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:01.393966 23388 ts_tablet_manager.cc:1367] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.311s user 0.031s sys 0.026s
I20240502 14:06:01.381203 23344 raft_consensus.cc:2454] T 55c964342c58447b955034291dc40cb6 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.400272 23388 raft_consensus.cc:348] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.401711 23388 raft_consensus.cc:374] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.401999 23388 raft_consensus.cc:729] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.402652 23388 consensus_queue.cc:260] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.414049 23391 tablet_bootstrap.cc:654] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
W20240502 14:06:01.414638 23313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 78378.4us
I20240502 14:06:01.417131 23388 ts_tablet_manager.cc:1398] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240502 14:06:01.419833 23388 tablet_bootstrap.cc:492] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:01.406008 23658 raft_consensus.cc:504] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.417691 23192 raft_consensus.cc:2454] T 4d5d4ce1413141d0a1a08c3397a8ea70 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.426959 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.444128 23389 tablet_bootstrap.cc:654] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
W20240502 14:06:01.449862 23773 log.cc:924] Time spent T 69a22b60b6bb46189f2c845b2d299a3f P 980e09c2959e408db93d72b7125ccf63: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
W20240502 14:06:01.451213 23765 log.cc:924] Time spent T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20240502 14:06:01.451723 23776 raft_consensus.cc:482] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.453689 23775 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.452163 23776 raft_consensus.cc:504] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.456411 23658 leader_election.cc:288] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.461089 23238 leader_election.cc:302] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.462117 23658 raft_consensus.cc:2781] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.462584 23658 raft_consensus.cc:686] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:01.463354 23658 consensus_queue.cc:237] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.464946 23776 leader_election.cc:288] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.469657 23391 tablet_bootstrap.cc:492] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.470088 23391 ts_tablet_manager.cc:1367] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.252s user 0.010s sys 0.000s
I20240502 14:06:01.472630 23391 raft_consensus.cc:348] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.473251 23391 raft_consensus.cc:374] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.473508 23391 raft_consensus.cc:729] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.476004 23388 tablet_bootstrap.cc:654] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.476253 23717 consensus_queue.cc:1035] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:06:01.477232 23238 leader_election.cc:341] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.454144 23775 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.480073 23775 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.481011 23641 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:01.489349 23775 consensus_queue.cc:1035] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:01.489547 23391 consensus_queue.cc:260] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.492381 23755 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.492764 23744 consensus_queue.cc:1035] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:01.492853 23755 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.494951 23755 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.492874 23388 tablet_bootstrap.cc:492] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:01.496634 23391 ts_tablet_manager.cc:1398] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20240502 14:06:01.496768 23388 ts_tablet_manager.cc:1367] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.002s
I20240502 14:06:01.497488 23391 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:01.499508 23388 raft_consensus.cc:348] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.500227 23388 raft_consensus.cc:374] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.500489 23388 raft_consensus.cc:729] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.501061 23388 consensus_queue.cc:260] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.503104 23389 tablet_bootstrap.cc:492] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:01.503554 23389 ts_tablet_manager.cc:1367] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.126s user 0.010s sys 0.002s
I20240502 14:06:01.504420 23391 tablet_bootstrap.cc:654] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.504958 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afca5b05bac0423ca02254a903b07eca" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.504665 23090 catalog_manager.cc:5484] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:01.505517 23346 raft_consensus.cc:3037] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.505841 23389 raft_consensus.cc:348] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.506474 23389 raft_consensus.cc:374] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.506711 23389 raft_consensus.cc:729] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.507323 23389 consensus_queue.cc:260] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.509436 23389 ts_tablet_manager.cc:1398] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:01.510273 23389 tablet_bootstrap.cc:492] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:01.511998 23346 raft_consensus.cc:2454] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.516638 23391 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.517040 23391 ts_tablet_manager.cc:1367] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20240502 14:06:01.520197 23388 ts_tablet_manager.cc:1398] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.023s user 0.001s sys 0.005s
I20240502 14:06:01.520212 23391 raft_consensus.cc:348] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.521070 23391 raft_consensus.cc:374] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.521160 23388 tablet_bootstrap.cc:492] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:01.521369 23391 raft_consensus.cc:729] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.522030 23391 consensus_queue.cc:260] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.523964 23391 ts_tablet_manager.cc:1398] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240502 14:06:01.524822 23391 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:01.528007 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.528632 23340 raft_consensus.cc:2454] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.529639 23391 tablet_bootstrap.cc:654] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.531399 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.532145 23269 raft_consensus.cc:2379] T 8f2a73e349254db1a61fc140ca00d99c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:06:01.533499 23389 tablet_bootstrap.cc:654] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.536921 23388 tablet_bootstrap.cc:654] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.537247 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.537869 23269 raft_consensus.cc:2454] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:06:01.540560 23391 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.540968 23391 ts_tablet_manager.cc:1367] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20240502 14:06:01.543134 23391 raft_consensus.cc:348] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.543780 23391 raft_consensus.cc:374] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.544026 23391 raft_consensus.cc:729] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.544692 23391 consensus_queue.cc:260] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.545943 23238 leader_election.cc:302] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:06:01.546675 23755 raft_consensus.cc:2781] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.547281 23755 raft_consensus.cc:686] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:01.547595 23391 ts_tablet_manager.cc:1398] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 14:06:01.548063 23755 consensus_queue.cc:237] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.548440 23391 tablet_bootstrap.cc:492] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:01.550500 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afca5b05bac0423ca02254a903b07eca" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:01.551085 23197 raft_consensus.cc:3037] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.552568 23388 tablet_bootstrap.cc:492] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:01.553033 23388 ts_tablet_manager.cc:1367] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.002s
I20240502 14:06:01.554013 23391 tablet_bootstrap.cc:654] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.555740 23388 raft_consensus.cc:348] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.556020 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.556401 23388 raft_consensus.cc:374] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.556428 23682 raft_consensus.cc:482] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.556797 23388 raft_consensus.cc:729] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.556875 23192 raft_consensus.cc:2454] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.556978 23682 raft_consensus.cc:504] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.558733 23389 tablet_bootstrap.cc:492] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:01.558930 23238 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:06:01.559151 23389 ts_tablet_manager.cc:1367] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.000s
I20240502 14:06:01.559433 23682 leader_election.cc:288] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.559646 23536 raft_consensus.cc:2781] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:01.560027 23536 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.560355 23536 raft_consensus.cc:3037] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:01.560833 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.561436 23340 raft_consensus.cc:2454] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:01.562474 23191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2a73e349254db1a61fc140ca00d99c" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.562475 23162 leader_election.cc:302] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.564247 23682 raft_consensus.cc:2781] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.564574 23682 raft_consensus.cc:482] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.564880 23682 raft_consensus.cc:3037] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.567121 23536 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.570012 23536 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.571383 23388 consensus_queue.cc:260] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.572192 23391 tablet_bootstrap.cc:492] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.571512 23682 raft_consensus.cc:504] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.572605 23391 ts_tablet_manager.cc:1367] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.006s
I20240502 14:06:01.572716 23389 raft_consensus.cc:348] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.573544 23389 raft_consensus.cc:374] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.573813 23389 raft_consensus.cc:729] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.572993 23388 ts_tablet_manager.cc:1398] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 14:06:01.574594 23682 leader_election.cc:288] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.574399 23389 consensus_queue.cc:260] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.574844 23388 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:01.574975 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.579783 23340 raft_consensus.cc:3037] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.575106 23775 raft_consensus.cc:482] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.588742 23777 consensus_queue.cc:1035] T 25d41b81c5154af7a4644e29408cb4cf P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:01.589232 23775 raft_consensus.cc:504] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.591538 23775 leader_election.cc:288] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.575909 23391 raft_consensus.cc:348] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.585731 23726 raft_consensus.cc:482] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.592180 23726 raft_consensus.cc:504] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.594187 23726 leader_election.cc:288] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.595479 23391 raft_consensus.cc:374] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.595767 23391 raft_consensus.cc:729] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.596324 23391 consensus_queue.cc:260] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.596905 23197 raft_consensus.cc:2454] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.598821 23391 ts_tablet_manager.cc:1398] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.026s user 0.003s sys 0.001s
I20240502 14:06:01.599725 23391 tablet_bootstrap.cc:492] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:01.599566 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b074ab1c0a041dabe1f6243776950a5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.600262 23346 raft_consensus.cc:2454] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:01.601385 23162 leader_election.cc:302] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.602644 23726 raft_consensus.cc:2781] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.603044 23731 raft_consensus.cc:482] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.603554 23731 raft_consensus.cc:504] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.604890 23391 tablet_bootstrap.cc:654] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.605331 23731 leader_election.cc:288] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.614988 23188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adefff536c04350a45b88a896f35226" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.587761 23340 raft_consensus.cc:2454] T 11d15424bb7b49cb899aba05cea68af0 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:01.585747 23388 tablet_bootstrap.cc:654] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.627250 23162 leader_election.cc:302] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.628710 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.629501 23310 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:06:01.586555 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.616215 23194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:01.630326 23774 raft_consensus.cc:2781] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:01.630479 23194 raft_consensus.cc:3037] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:01.630740 23774 raft_consensus.cc:482] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.631093 23774 raft_consensus.cc:3037] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:01.633209 23313 leader_election.cc:302] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.618103 23344 raft_consensus.cc:3037] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.637615 23774 raft_consensus.cc:504] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.640504 23682 raft_consensus.cc:2781] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.641254 23682 raft_consensus.cc:686] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:01.642499 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:01.643234 23194 raft_consensus.cc:2454] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 2.
I20240502 14:06:01.602998 23726 raft_consensus.cc:482] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.587237 23389 ts_tablet_manager.cc:1398] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20240502 14:06:01.618669 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eda455829f14de7865c64312c2bdb9d" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.619760 23238 leader_election.cc:302] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:01.644029 23726 raft_consensus.cc:3037] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.644893 23536 raft_consensus.cc:2781] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.645893 23536 raft_consensus.cc:686] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:01.646132 23389 tablet_bootstrap.cc:492] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:01.646831 23536 consensus_queue.cc:237] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.647966 23525 raft_consensus.cc:2726] T 8f2a73e349254db1a61fc140ca00d99c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.649317 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b074ab1c0a041dabe1f6243776950a5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.650748 23744 consensus_queue.cc:1035] T 8f2a73e349254db1a61fc140ca00d99c P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:01.654215 23389 tablet_bootstrap.cc:654] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.626113 23391 tablet_bootstrap.cc:492] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.643646 23774 leader_election.cc:288] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 2 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.655516 23391 ts_tablet_manager.cc:1367] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.016s
I20240502 14:06:01.658191 23391 raft_consensus.cc:348] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.658838 23391 raft_consensus.cc:374] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.659088 23391 raft_consensus.cc:729] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.659649 23391 consensus_queue.cc:260] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.661188 23391 ts_tablet_manager.cc:1398] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:06:01.661347 23192 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 2: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:06:01.662068 23391 tablet_bootstrap.cc:492] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
W20240502 14:06:01.663631 23313 leader_election.cc:341] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.663408 23682 consensus_queue.cc:237] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.667707 23268 raft_consensus.cc:3037] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.669447 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adefff536c04350a45b88a896f35226" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.670087 23267 raft_consensus.cc:2454] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 0.
I20240502 14:06:01.626073 23092 catalog_manager.cc:5484] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:01.682104 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:01.682883 23192 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 2: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:06:01.683761 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
W20240502 14:06:01.679001 23160 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.21.129.62:37293, user_credentials={real_user=slave}} blocked reactor thread for 57113us
I20240502 14:06:01.684523 23267 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 2: Already voted for candidate 82490d85cd3d4a1180dc3ecabdd88598 in this term.
I20240502 14:06:01.690886 23238 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.691593 23729 raft_consensus.cc:2781] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:01.692056 23729 raft_consensus.cc:686] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [term 2 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
W20240502 14:06:01.693168 23238 leader_election.cc:341] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.692801 23729 consensus_queue.cc:237] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.681509 23726 raft_consensus.cc:504] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:01.695065 23159 leader_election.cc:341] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:01.699539 23159 leader_election.cc:341] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:01.701756 23159 leader_election.cc:341] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.706632 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b074ab1c0a041dabe1f6243776950a5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.727617 23343 raft_consensus.cc:3037] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.734398 23343 raft_consensus.cc:2454] T 0b074ab1c0a041dabe1f6243776950a5 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:01.709332 23726 leader_election.cc:288] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.710675 23658 raft_consensus.cc:482] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.735559 23162 leader_election.cc:302] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.735481 23658 raft_consensus.cc:504] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.736428 23726 raft_consensus.cc:2781] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.737041 23726 raft_consensus.cc:686] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:01.737437 23658 leader_election.cc:288] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.738422 23310 leader_election.cc:302] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.739205 23774 raft_consensus.cc:2781] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.739604 23774 raft_consensus.cc:482] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.739935 23774 raft_consensus.cc:3037] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.716717 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eda455829f14de7865c64312c2bdb9d" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.720139 23605 raft_consensus.cc:482] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.741789 23605 raft_consensus.cc:504] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.743651 23605 leader_election.cc:288] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.695323 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994cb5cd6fe74f2293ca092bc550ffb3" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.745131 23344 raft_consensus.cc:2379] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 82490d85cd3d4a1180dc3ecabdd88598 in current term 2: Already voted for candidate 980e09c2959e408db93d72b7125ccf63 in this term.
I20240502 14:06:01.745911 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:06:01.747133 23162 leader_election.cc:341] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.748703 23090 catalog_manager.cc:5484] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:01.750021 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.755595 23388 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:01.756250 23388 ts_tablet_manager.cc:1367] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.182s user 0.014s sys 0.009s
I20240502 14:06:01.757380 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b074ab1c0a041dabe1f6243776950a5" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:01.758453 23388 raft_consensus.cc:348] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.737859 23726 consensus_queue.cc:237] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.763829 23774 raft_consensus.cc:504] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:01.765259 23159 leader_election.cc:341] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.759121 23388 raft_consensus.cc:374] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.765969 23774 leader_election.cc:288] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.766548 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.769876 23722 consensus_queue.cc:1035] T cd50bda89ed043c4900bd94025639e77 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:01.782513 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adefff536c04350a45b88a896f35226" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:01.769954 23310 leader_election.cc:302] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.783455 23722 raft_consensus.cc:2726] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 14:06:01.783715 23313 leader_election.cc:341] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:06:01.787325 23238 leader_election.cc:341] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.787765 23238 leader_election.cc:302] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.771973 23391 tablet_bootstrap.cc:654] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.788939 23536 raft_consensus.cc:2726] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.790618 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adefff536c04350a45b88a896f35226" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:01.791287 23269 raft_consensus.cc:3037] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.773981 23389 tablet_bootstrap.cc:492] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:01.792652 23389 ts_tablet_manager.cc:1367] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.147s user 0.015s sys 0.000s
I20240502 14:06:01.790900 23089 catalog_manager.cc:5484] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:01.781370 23388 raft_consensus.cc:729] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.782575 23267 raft_consensus.cc:2454] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:01.770143 23755 raft_consensus.cc:482] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.793962 23755 raft_consensus.cc:504] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.796506 23755 leader_election.cc:288] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.800091 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd50bda89ed043c4900bd94025639e77" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.800820 23269 raft_consensus.cc:2454] T 6adefff536c04350a45b88a896f35226 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980e09c2959e408db93d72b7125ccf63 in term 1.
I20240502 14:06:01.802747 23389 raft_consensus.cc:348] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.803494 23389 raft_consensus.cc:374] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.803766 23389 raft_consensus.cc:729] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.802793 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb27a4bdc15414cb260ffc2b673b5ab" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:01.804566 23388 consensus_queue.cc:260] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.806712 23388 ts_tablet_manager.cc:1398] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20240502 14:06:01.807611 23388 tablet_bootstrap.cc:492] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:01.810338 23159 leader_election.cc:302] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:01.804894 23192 raft_consensus.cc:2454] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:01.812055 23238 leader_election.cc:302] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.813060 23753 raft_consensus.cc:2781] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.812860 23389 consensus_queue.cc:260] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.813441 23753 raft_consensus.cc:482] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.813799 23753 raft_consensus.cc:3037] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.814954 23389 ts_tablet_manager.cc:1398] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240502 14:06:01.815779 23389 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:01.819020 23388 tablet_bootstrap.cc:654] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.821087 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb27a4bdc15414cb260ffc2b673b5ab" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:01.821799 23344 raft_consensus.cc:2454] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:01.822398 23238 leader_election.cc:302] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.823544 23726 raft_consensus.cc:2781] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.824232 23726 raft_consensus.cc:482] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.822882 23753 raft_consensus.cc:504] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.824599 23726 raft_consensus.cc:3037] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.826102 23310 leader_election.cc:302] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters: cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:01.826915 23722 raft_consensus.cc:2781] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.827406 23722 raft_consensus.cc:686] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Becoming Leader. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Running, Role: LEADER
I20240502 14:06:01.826305 23753 leader_election.cc:288] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.827648 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb27a4bdc15414cb260ffc2b673b5ab" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:01.828246 23722 consensus_queue.cc:237] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.828814 23192 raft_consensus.cc:3037] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.834590 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb27a4bdc15414cb260ffc2b673b5ab" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.835667 23343 raft_consensus.cc:3037] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.837594 23389 tablet_bootstrap.cc:654] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.842393 23192 raft_consensus.cc:2454] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.856889 23238 leader_election.cc:302] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:01.864660 23536 raft_consensus.cc:2781] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.865118 23536 raft_consensus.cc:686] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:01.865877 23536 consensus_queue.cc:237] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.878865 23343 raft_consensus.cc:2454] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:01.880856 23726 raft_consensus.cc:504] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.900275 23658 raft_consensus.cc:2726] T cd50bda89ed043c4900bd94025639e77 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:01.904664 23391 tablet_bootstrap.cc:492] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:01.905123 23391 ts_tablet_manager.cc:1367] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.243s user 0.008s sys 0.002s
I20240502 14:06:01.906739 23092 catalog_manager.cc:5484] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:01.908448 23092 catalog_manager.cc:5484] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:01.909616 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:01.907141 23726 leader_election.cc:288] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.921224 23268 raft_consensus.cc:3037] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.912789 23389 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:01.920238 23391 raft_consensus.cc:348] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.922240 23389 ts_tablet_manager.cc:1367] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.000s
I20240502 14:06:01.922273 23391 raft_consensus.cc:374] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.922645 23391 raft_consensus.cc:729] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.923427 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:01.923239 23391 consensus_queue.cc:260] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.925235 23389 raft_consensus.cc:348] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.915778 23388 tablet_bootstrap.cc:492] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:01.925755 23391 ts_tablet_manager.cc:1398] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 14:06:01.926193 23388 ts_tablet_manager.cc:1367] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.119s user 0.013s sys 0.005s
I20240502 14:06:01.930691 23388 raft_consensus.cc:348] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.931349 23388 raft_consensus.cc:374] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.931588 23388 raft_consensus.cc:729] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.925884 23389 raft_consensus.cc:374] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.932096 23389 raft_consensus.cc:729] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.934536 23268 raft_consensus.cc:2454] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
W20240502 14:06:01.928279 23162 leader_election.cc:341] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.938179 23159 leader_election.cc:302] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:01.939285 23605 raft_consensus.cc:2781] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:01.939718 23605 raft_consensus.cc:686] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:01.940459 23605 consensus_queue.cc:237] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.932690 23389 consensus_queue.cc:260] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.943029 23389 ts_tablet_manager.cc:1398] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.020s user 0.003s sys 0.002s
I20240502 14:06:01.944255 23388 consensus_queue.cc:260] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.946192 23388 ts_tablet_manager.cc:1398] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240502 14:06:01.947002 23388 tablet_bootstrap.cc:492] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:01.947278 23389 tablet_bootstrap.cc:492] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:01.947516 23774 raft_consensus.cc:482] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.947932 23774 raft_consensus.cc:504] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.949651 23774 leader_election.cc:288] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:01.951050 23753 raft_consensus.cc:482] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.951536 23753 raft_consensus.cc:504] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:01.954165 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:01.954588 23391 tablet_bootstrap.cc:492] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:01.954802 23753 leader_election.cc:288] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:01.954986 23267 raft_consensus.cc:2379] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:06:01.956205 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:06:01.957509 23238 leader_election.cc:341] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.971500 23092 catalog_manager.cc:5484] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 2, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 2 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
W20240502 14:06:01.984104 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:06:02.001003 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:06:02.002190 23238 leader_election.cc:341] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.002574 23238 leader_election.cc:302] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:02.003696 23641 raft_consensus.cc:2726] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:01.990141 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6132f4449de74e15a3cbe93f03a74b71" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:02.010123 23188 raft_consensus.cc:3037] T 55c964342c58447b955034291dc40cb6 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.014055 23313 leader_election.cc:302] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:02.011252 23658 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 14:06:02.017560 23388 tablet_bootstrap.cc:654] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.018378 23717 raft_consensus.cc:3037] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.018767 23389 tablet_bootstrap.cc:654] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.047565 23089 catalog_manager.cc:5484] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:02.049291 23089 catalog_manager.cc:5484] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:02.056815 23717 raft_consensus.cc:2726] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:02.069602 23391 tablet_bootstrap.cc:654] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.060339 23641 consensus_queue.cc:1035] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 14:06:02.067750 23605 raft_consensus.cc:482] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.081140 23755 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240502 14:06:02.092633 23388 tablet_bootstrap.cc:492] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:02.076179 23658 consensus_queue.cc:1035] T 55c964342c58447b955034291dc40cb6 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240502 14:06:02.093137 23389 tablet_bootstrap.cc:492] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:02.093573 23389 ts_tablet_manager.cc:1367] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.150s user 0.007s sys 0.020s
I20240502 14:06:02.094297 23388 ts_tablet_manager.cc:1367] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.147s user 0.021s sys 0.017s
I20240502 14:06:02.097446 23388 raft_consensus.cc:348] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.098686 23389 raft_consensus.cc:348] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.099403 23389 raft_consensus.cc:374] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.099694 23389 raft_consensus.cc:729] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.100493 23389 consensus_queue.cc:260] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.102587 23389 ts_tablet_manager.cc:1398] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240502 14:06:02.103624 23389 tablet_bootstrap.cc:492] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:02.077896 23605 raft_consensus.cc:504] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.099411 23388 raft_consensus.cc:374] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.105271 23388 raft_consensus.cc:729] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.105796 23388 consensus_queue.cc:260] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.110106 23091 catalog_manager.cc:5484] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 980e09c2959e408db93d72b7125ccf63 (127.21.129.3). New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:02.116672 23388 ts_tablet_manager.cc:1398] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240502 14:06:02.116057 23092 catalog_manager.cc:5484] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:02.121290 23388 tablet_bootstrap.cc:492] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:02.165925 23389 tablet_bootstrap.cc:654] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.209342 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec446373869d4c05ad8c5807b73740bf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
W20240502 14:06:02.211180 23159 leader_election.cc:341] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.211522 23605 leader_election.cc:288] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.222999 23726 consensus_queue.cc:1035] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:02.226094 23159 consensus_peers.cc:485] T 11d15424bb7b49cb899aba05cea68af0 P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Couldn't send request to peer 82490d85cd3d4a1180dc3ecabdd88598. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:02.229198 23391 tablet_bootstrap.cc:492] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:02.229652 23391 ts_tablet_manager.cc:1367] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.275s user 0.022s sys 0.016s
I20240502 14:06:02.234941 23391 raft_consensus.cc:348] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.235584 23391 raft_consensus.cc:374] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.235846 23391 raft_consensus.cc:729] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.236438 23391 consensus_queue.cc:260] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.237973 23391 ts_tablet_manager.cc:1398] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240502 14:06:02.238843 23391 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:02.241585 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec446373869d4c05ad8c5807b73740bf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:02.242192 23340 raft_consensus.cc:2454] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:02.249522 23525 consensus_queue.cc:1035] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.252352 23162 leader_election.cc:302] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:06:02.253468 23816 raft_consensus.cc:2781] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:02.253827 23816 raft_consensus.cc:482] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:02.254132 23816 raft_consensus.cc:3037] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.265822 23391 tablet_bootstrap.cc:654] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.272213 23388 tablet_bootstrap.cc:654] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
W20240502 14:06:02.280321 23313 consensus_peers.cc:485] T 6adefff536c04350a45b88a896f35226 P 980e09c2959e408db93d72b7125ccf63 -> Peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Couldn't send request to peer cc4007c6e59b4361a155c5e8a79bbfb7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:02.297026 23388 tablet_bootstrap.cc:492] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:02.297472 23388 ts_tablet_manager.cc:1367] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.176s user 0.003s sys 0.025s
I20240502 14:06:02.299932 23389 tablet_bootstrap.cc:492] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:02.299739 23388 raft_consensus.cc:348] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.300391 23389 ts_tablet_manager.cc:1367] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.197s user 0.010s sys 0.001s
I20240502 14:06:02.303159 23268 raft_consensus.cc:3037] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.303027 23389 raft_consensus.cc:348] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.305878 23389 raft_consensus.cc:374] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.306772 23389 raft_consensus.cc:729] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.309846 23191 raft_consensus.cc:1261] T 994cb5cd6fe74f2293ca092bc550ffb3 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 2 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:02.315044 23536 raft_consensus.cc:482] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.315529 23536 raft_consensus.cc:504] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.320035 23536 leader_election.cc:288] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.321211 23817 consensus_queue.cc:1035] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.314723 23816 raft_consensus.cc:504] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.333117 23682 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.334048 23817 consensus_queue.cc:1035] T 0b074ab1c0a041dabe1f6243776950a5 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.338631 23816 leader_election.cc:288] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.300451 23388 raft_consensus.cc:374] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.340696 23388 raft_consensus.cc:729] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.341387 23388 consensus_queue.cc:260] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.342329 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec446373869d4c05ad8c5807b73740bf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:02.307395 23389 consensus_queue.cc:260] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.345822 23389 ts_tablet_manager.cc:1398] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20240502 14:06:02.348201 23389 tablet_bootstrap.cc:492] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:02.347342 23388 ts_tablet_manager.cc:1398] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.050s user 0.004s sys 0.003s
I20240502 14:06:02.350011 23388 tablet_bootstrap.cc:492] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
I20240502 14:06:02.350278 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec446373869d4c05ad8c5807b73740bf" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:02.353173 23346 raft_consensus.cc:3037] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:06:02.354574 23159 leader_election.cc:341] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.360095 23536 consensus_queue.cc:1035] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.363257 23726 raft_consensus.cc:482] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.363142 23755 consensus_queue.cc:1035] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:02.363888 23726 raft_consensus.cc:504] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.362079 23605 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:02.374086 23726 leader_election.cc:288] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.375624 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eda455829f14de7865c64312c2bdb9d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:02.376235 23268 raft_consensus.cc:2454] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:02.375577 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eda455829f14de7865c64312c2bdb9d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:02.377450 23343 raft_consensus.cc:2454] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:02.378597 23159 leader_election.cc:302] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:02.378985 23391 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:02.379426 23391 ts_tablet_manager.cc:1367] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.141s user 0.049s sys 0.023s
I20240502 14:06:02.394093 23391 raft_consensus.cc:348] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.394734 23391 raft_consensus.cc:374] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.394994 23391 raft_consensus.cc:729] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.376407 23389 tablet_bootstrap.cc:654] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.395576 23391 consensus_queue.cc:260] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:02.411409 23238 consensus_peers.cc:485] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Couldn't send request to peer 980e09c2959e408db93d72b7125ccf63. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:02.415800 23188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b074ab1c0a041dabe1f6243776950a5" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:02.415989 23344 raft_consensus.cc:1261] T 994cb5cd6fe74f2293ca092bc550ffb3 P 980e09c2959e408db93d72b7125ccf63 [term 2 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:02.418897 23536 consensus_queue.cc:1035] T 994cb5cd6fe74f2293ca092bc550ffb3 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.402863 23816 raft_consensus.cc:2781] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:02.420778 23816 raft_consensus.cc:482] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:02.422889 23816 raft_consensus.cc:3037] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.427605 23391 ts_tablet_manager.cc:1398] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.048s user 0.010s sys 0.000s
I20240502 14:06:02.429304 23391 tablet_bootstrap.cc:492] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:02.444133 23388 tablet_bootstrap.cc:654] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.420482 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b074ab1c0a041dabe1f6243776950a5" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:02.473783 23725 raft_consensus.cc:482] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.512121 23238 leader_election.cc:302] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:02.511587 23346 raft_consensus.cc:2454] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:02.529189 23658 raft_consensus.cc:2726] T 0b074ab1c0a041dabe1f6243776950a5 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:02.501698 23725 raft_consensus.cc:504] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.533741 23658 raft_consensus.cc:482] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.552582 23725 leader_election.cc:288] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.554414 23391 tablet_bootstrap.cc:654] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.501365 23816 raft_consensus.cc:504] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.569377 23389 tablet_bootstrap.cc:492] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:02.666514 23389 ts_tablet_manager.cc:1367] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.318s user 0.067s sys 0.052s
I20240502 14:06:02.656037 23388 tablet_bootstrap.cc:492] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:02.667845 23388 ts_tablet_manager.cc:1367] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.318s user 0.030s sys 0.011s
I20240502 14:06:02.669221 23389 raft_consensus.cc:348] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.669945 23389 raft_consensus.cc:374] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.670195 23389 raft_consensus.cc:729] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.577262 23162 leader_election.cc:302] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598
I20240502 14:06:02.671218 23817 raft_consensus.cc:2781] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:02.671711 23817 raft_consensus.cc:686] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:02.671063 23389 consensus_queue.cc:260] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.670238 23388 raft_consensus.cc:348] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:02.671375 23162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 106352us
I20240502 14:06:02.571080 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:02.673187 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eda455829f14de7865c64312c2bdb9d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:02.673724 23267 raft_consensus.cc:3037] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.556340 23658 raft_consensus.cc:504] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.675257 23389 ts_tablet_manager.cc:1398] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:06:02.676079 23389 tablet_bootstrap.cc:492] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:02.672806 23388 raft_consensus.cc:374] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.676725 23388 raft_consensus.cc:729] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.677392 23388 consensus_queue.cc:260] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.672508 23817 consensus_queue.cc:237] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.666990 23816 leader_election.cc:288] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.650372 23729 consensus_queue.cc:1035] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20240502 14:06:02.701931 23388 ts_tablet_manager.cc:1398] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.034s user 0.003s sys 0.003s
W20240502 14:06:02.701841 23159 leader_election.cc:341] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.695744 23658 leader_election.cc:288] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.703006 23388 tablet_bootstrap.cc:492] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598: Bootstrap starting.
W20240502 14:06:02.706786 23313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 142806us
I20240502 14:06:02.715479 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e187bdb41455fbddec2888eb6efa8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:02.766676 23816 raft_consensus.cc:482] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.767184 23816 raft_consensus.cc:504] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.769040 23816 leader_election.cc:288] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.769094 23389 tablet_bootstrap.cc:654] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.779881 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:02.780627 23270 raft_consensus.cc:2454] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:02.782075 23159 leader_election.cc:302] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:02.718309 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:02.718468 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e187bdb41455fbddec2888eb6efa8" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
W20240502 14:06:02.784978 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 75818us
I20240502 14:06:02.705752 23836 consensus_queue.cc:1035] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240502 14:06:02.777459 23842 raft_consensus.cc:482] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.793337 23842 raft_consensus.cc:504] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.793794 23745 raft_consensus.cc:2781] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 14:06:02.807309 23238 leader_election.cc:341] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.800577 23745 raft_consensus.cc:482] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:02.815999 23536 raft_consensus.cc:482] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.816548 23536 raft_consensus.cc:504] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.820106 23391 tablet_bootstrap.cc:492] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:02.820667 23391 ts_tablet_manager.cc:1367] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.392s user 0.021s sys 0.019s
W20240502 14:06:02.822897 23238 leader_election.cc:341] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.824497 23345 raft_consensus.cc:3037] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.826524 23388 tablet_bootstrap.cc:654] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.811378 23745 raft_consensus.cc:3037] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.837011 23536 leader_election.cc:288] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.823500 23238 leader_election.cc:302] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:02.838685 23536 raft_consensus.cc:2726] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:02.867929 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eda455829f14de7865c64312c2bdb9d" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:02.868644 23341 raft_consensus.cc:3037] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:06:02.887718 23845 log.cc:924] Time spent T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240502 14:06:02.890995 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:02.891713 23272 raft_consensus.cc:3037] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.891991 23389 tablet_bootstrap.cc:492] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:02.892449 23389 ts_tablet_manager.cc:1367] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.217s user 0.018s sys 0.015s
I20240502 14:06:02.896978 23391 raft_consensus.cc:348] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.897712 23391 raft_consensus.cc:374] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.897974 23391 raft_consensus.cc:729] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.898555 23391 consensus_queue.cc:260] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.889514 23745 raft_consensus.cc:504] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.903069 23745 leader_election.cc:288] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:02.904731 23391 ts_tablet_manager.cc:1398] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.084s user 0.006s sys 0.000s
I20240502 14:06:02.860016 23267 raft_consensus.cc:2454] T 1eda455829f14de7865c64312c2bdb9d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:02.894989 23389 raft_consensus.cc:348] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.907729 23159 leader_election.cc:302] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:02.910187 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:02.910786 23269 raft_consensus.cc:3037] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.911250 23391 tablet_bootstrap.cc:492] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7: Bootstrap starting.
I20240502 14:06:02.917308 23745 raft_consensus.cc:2781] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:02.917996 23269 raft_consensus.cc:2454] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:02.907877 23389 raft_consensus.cc:374] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.921106 23159 leader_election.cc:302] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:02.922145 23745 raft_consensus.cc:686] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:02.922299 23605 raft_consensus.cc:2781] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:02.918535 23842 leader_election.cc:288] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:02.923377 23389 raft_consensus.cc:729] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.924078 23605 raft_consensus.cc:686] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:02.926785 23605 consensus_queue.cc:237] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.928148 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:02.928378 23389 consensus_queue.cc:260] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.942811 23389 ts_tablet_manager.cc:1398] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20240502 14:06:02.924772 23745 consensus_queue.cc:237] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:02.945948 23092 catalog_manager.cc:5484] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:02.953821 23389 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
W20240502 14:06:02.983289 23162 leader_election.cc:341] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.999764 23391 tablet_bootstrap.cc:654] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:03.005255 23388 tablet_bootstrap.cc:492] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598: No bootstrap required, opened a new log
I20240502 14:06:03.005723 23388 ts_tablet_manager.cc:1367] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent bootstrapping tablet: real 0.303s user 0.020s sys 0.051s
I20240502 14:06:02.982389 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4ce1413141d0a1a08c3397a8ea70" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:03.009197 23658 raft_consensus.cc:482] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:03.018440 23341 raft_consensus.cc:2454] T 1eda455829f14de7865c64312c2bdb9d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:03.001930 23162 leader_election.cc:302] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:03.012176 23726 raft_consensus.cc:482] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:03.040727 23388 raft_consensus.cc:348] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.057741 23388 raft_consensus.cc:374] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:03.058030 23388 raft_consensus.cc:729] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Initialized, Role: FOLLOWER
W20240502 14:06:03.060083 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 85795.4us
I20240502 14:06:03.064067 23388 consensus_queue.cc:260] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.022073 23191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:03.029548 23340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:03.027844 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d15424bb7b49cb899aba05cea68af0" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
W20240502 14:06:03.091974 23141 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:06:03.098927 23388 ts_tablet_manager.cc:1398] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598: Time spent starting tablet: real 0.093s user 0.008s sys 0.002s
I20240502 14:06:03.071492 23745 raft_consensus.cc:2726] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:03.059628 23658 raft_consensus.cc:504] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.104199 23658 leader_election.cc:288] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:03.105603 23389 tablet_bootstrap.cc:654] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
W20240502 14:06:03.086915 23162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 98492.6us
W20240502 14:06:03.115666 23162 leader_election.cc:341] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:03.064965 23726 raft_consensus.cc:504] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.148289 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec446373869d4c05ad8c5807b73740bf" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:03.148458 23726 leader_election.cc:288] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:03.154803 23238 leader_election.cc:302] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:03.221349 23313 leader_election.cc:302] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:03.225888 23841 raft_consensus.cc:2726] T 4d5d4ce1413141d0a1a08c3397a8ea70 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:03.165233 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec446373869d4c05ad8c5807b73740bf" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:03.228222 23343 raft_consensus.cc:2379] T ec446373869d4c05ad8c5807b73740bf P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82490d85cd3d4a1180dc3ecabdd88598 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:06:03.227056 23389 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:03.230086 23389 ts_tablet_manager.cc:1367] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.277s user 0.016s sys 0.003s
I20240502 14:06:03.232586 23391 tablet_bootstrap.cc:492] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7: No bootstrap required, opened a new log
I20240502 14:06:03.233033 23391 ts_tablet_manager.cc:1367] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent bootstrapping tablet: real 0.322s user 0.015s sys 0.012s
I20240502 14:06:03.235388 23391 raft_consensus.cc:348] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.236048 23391 raft_consensus.cc:374] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:03.236327 23391 raft_consensus.cc:729] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Initialized, Role: FOLLOWER
I20240502 14:06:03.236968 23391 consensus_queue.cc:260] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.239267 23391 ts_tablet_manager.cc:1398] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:06:03.206964 23265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e187bdb41455fbddec2888eb6efa8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:03.240087 23265 raft_consensus.cc:2454] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:03.241115 23159 leader_election.cc:302] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:03.241897 23846 raft_consensus.cc:2726] T 11d15424bb7b49cb899aba05cea68af0 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:06:03.242028 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 87300.3us
I20240502 14:06:03.243708 23189 raft_consensus.cc:3037] T 6adefff536c04350a45b88a896f35226 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.243839 23605 raft_consensus.cc:2781] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:03.244220 23605 raft_consensus.cc:482] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:03.244480 23605 raft_consensus.cc:3037] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.248310 23389 raft_consensus.cc:348] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.249040 23389 raft_consensus.cc:374] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:03.249352 23389 raft_consensus.cc:729] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:03.250152 23389 consensus_queue.cc:260] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.254781 23605 raft_consensus.cc:504] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.258332 23389 ts_tablet_manager.cc:1398] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.028s user 0.004s sys 0.001s
I20240502 14:06:03.260113 23389 tablet_bootstrap.cc:492] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63: Bootstrap starting.
I20240502 14:06:03.260840 23605 leader_election.cc:288] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:03.263300 23265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e187bdb41455fbddec2888eb6efa8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598"
I20240502 14:06:03.263948 23265 raft_consensus.cc:3037] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.266165 23389 tablet_bootstrap.cc:654] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:03.270335 23265 raft_consensus.cc:2454] T ff7e187bdb41455fbddec2888eb6efa8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:03.278013 23846 raft_consensus.cc:482] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:03.278484 23846 raft_consensus.cc:504] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.280398 23846 leader_election.cc:288] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:03.291515 23238 leader_election.cc:302] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598; no voters: 980e09c2959e408db93d72b7125ccf63, cc4007c6e59b4361a155c5e8a79bbfb7
I20240502 14:06:03.291817 23159 leader_election.cc:302] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters:
I20240502 14:06:03.292898 23729 raft_consensus.cc:3037] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.301975 23729 raft_consensus.cc:2726] T ec446373869d4c05ad8c5807b73740bf P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:03.306520 23726 raft_consensus.cc:2781] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:03.307018 23726 raft_consensus.cc:686] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Becoming Leader. State: Replica: cc4007c6e59b4361a155c5e8a79bbfb7, State: Running, Role: LEADER
I20240502 14:06:03.307842 23726 consensus_queue.cc:237] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.315845 23605 consensus_queue.cc:1035] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:03.318797 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:03.319453 23193 raft_consensus.cc:2454] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 0.
I20240502 14:06:03.331591 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:03.332396 23525 raft_consensus.cc:482] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:03.332819 23525 raft_consensus.cc:504] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
W20240502 14:06:03.333515 23162 leader_election.cc:341] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:03.334996 23525 leader_election.cc:288] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:03.336503 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" is_pre_election: true
I20240502 14:06:03.339726 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ebc495868467d95f7e6b636732483" candidate_uuid: "980e09c2959e408db93d72b7125ccf63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82490d85cd3d4a1180dc3ecabdd88598" is_pre_election: true
I20240502 14:06:03.340595 23272 raft_consensus.cc:2379] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 980e09c2959e408db93d72b7125ccf63 in current term 1: Already voted for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in this term.
I20240502 14:06:03.341675 23310 leader_election.cc:302] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980e09c2959e408db93d72b7125ccf63; no voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7
W20240502 14:06:03.346161 23238 leader_election.cc:341] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:03.348981 23841 raft_consensus.cc:3037] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.353456 23238 leader_election.cc:302] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, cc4007c6e59b4361a155c5e8a79bbfb7; no voters: 980e09c2959e408db93d72b7125ccf63
I20240502 14:06:03.354137 23658 raft_consensus.cc:2781] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:03.354483 23658 raft_consensus.cc:482] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:03.354851 23658 raft_consensus.cc:3037] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.355260 23841 raft_consensus.cc:2726] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:03.356045 23343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e187bdb41455fbddec2888eb6efa8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63" is_pre_election: true
I20240502 14:06:03.356643 23343 raft_consensus.cc:2454] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 0.
I20240502 14:06:03.356995 23389 tablet_bootstrap.cc:492] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63: No bootstrap required, opened a new log
I20240502 14:06:03.357409 23389 ts_tablet_manager.cc:1367] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63: Time spent bootstrapping tablet: real 0.097s user 0.032s sys 0.034s
I20240502 14:06:03.360008 23389 raft_consensus.cc:348] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.360670 23389 raft_consensus.cc:374] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:03.360458 23658 raft_consensus.cc:504] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.360949 23389 raft_consensus.cc:729] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980e09c2959e408db93d72b7125ccf63, State: Initialized, Role: FOLLOWER
I20240502 14:06:03.361569 23389 consensus_queue.cc:260] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.362221 23658 leader_election.cc:288] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Requested vote from peers cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171), 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:03.363606 23389 ts_tablet_manager.cc:1398] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:03.408841 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e187bdb41455fbddec2888eb6efa8" candidate_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:03.413126 23344 raft_consensus.cc:3037] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.429998 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7"
I20240502 14:06:03.437800 23863 consensus_queue.cc:1035] T ec446373869d4c05ad8c5807b73740bf P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:03.441690 23344 raft_consensus.cc:2454] T ff7e187bdb41455fbddec2888eb6efa8 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4007c6e59b4361a155c5e8a79bbfb7 in term 1.
I20240502 14:06:03.451848 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7635b879734a9e9c6a7f8c154251f1" candidate_uuid: "82490d85cd3d4a1180dc3ecabdd88598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980e09c2959e408db93d72b7125ccf63"
I20240502 14:06:03.452522 23339 raft_consensus.cc:3037] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.455497 23189 raft_consensus.cc:3037] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:03.465736 23339 raft_consensus.cc:2454] T fa7635b879734a9e9c6a7f8c154251f1 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:03.470461 23189 raft_consensus.cc:2454] T fa7635b879734a9e9c6a7f8c154251f1 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82490d85cd3d4a1180dc3ecabdd88598 in term 1.
I20240502 14:06:03.484083 23238 leader_election.cc:302] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82490d85cd3d4a1180dc3ecabdd88598, 980e09c2959e408db93d72b7125ccf63; no voters:
I20240502 14:06:03.485165 23536 raft_consensus.cc:2781] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:03.485873 23536 raft_consensus.cc:686] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Becoming Leader. State: Replica: 82490d85cd3d4a1180dc3ecabdd88598, State: Running, Role: LEADER
I20240502 14:06:03.486930 23536 consensus_queue.cc:237] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } }
I20240502 14:06:03.506642 23092 catalog_manager.cc:5484] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:03.514986 23605 consensus_queue.cc:1035] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:03.521592 23605 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:03.523437 23092 catalog_manager.cc:5484] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:03.537523 23726 consensus_queue.cc:1035] T 1eda455829f14de7865c64312c2bdb9d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:03.560729 23817 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:03.563694 23089 catalog_manager.cc:5484] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2). New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:03.587891 23089 catalog_manager.cc:5484] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1). New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:03.738430 23605 consensus_queue.cc:1035] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:03.757308 23726 consensus_queue.cc:1035] T ff7e187bdb41455fbddec2888eb6efa8 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:03.948436 23658 consensus_queue.cc:1035] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:03.961135 23729 consensus_queue.cc:1035] T fa7635b879734a9e9c6a7f8c154251f1 P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:04.508919 22020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:06:04.538561 22020 server_base.cc:1034] running on GCE node
W20240502 14:06:04.538765 23882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:04.539659 23883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:04.541178 23885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:04.545051 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:04.545276 22020 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:04.545457 22020 hybrid_clock.cc:648] HybridClock initialized: now 1714658764545440 us; error 0 us; skew 500 ppm
I20240502 14:06:04.546152 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:04.550037 22020 webserver.cc:460] Webserver started at http://127.21.129.4:37433/ using document root <none> and password file <none>
I20240502 14:06:04.551559 22020 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:04.551829 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:04.552235 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:04.554039 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-3-root/instance:
uuid: "67a108c8df8d451599ff168ac618d878"
format_stamp: "Formatted at 2024-05-02 14:06:04 on dist-test-slave-bdl7"
I20240502 14:06:04.564327 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.007s
I20240502 14:06:04.568746 23890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:04.569633 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:06:04.570005 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-3-root
uuid: "67a108c8df8d451599ff168ac618d878"
format_stamp: "Formatted at 2024-05-02 14:06:04 on dist-test-slave-bdl7"
I20240502 14:06:04.570305 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:04.594497 22020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:04.595778 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:04.596946 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:04.599102 22020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:04.599320 22020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:04.599515 22020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:04.599658 22020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:04.660784 22020 rpc_server.cc:307] RPC server started. Bound to: 127.21.129.4:46761
I20240502 14:06:04.660863 23953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.129.4:46761 every 8 connection(s)
I20240502 14:06:04.664925 22020 auto_leader_rebalancer-test.cc:405] add a tserver: 67a108c8df8d451599ff168ac618d878
I20240502 14:06:04.676779 23954 heartbeater.cc:347] Connected to a master server at 127.21.129.62:37293
I20240502 14:06:04.677167 23954 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:04.677835 23954 heartbeater.cc:511] Master 127.21.129.62:37293 requested a full tablet report, sending...
I20240502 14:06:04.679492 23092 ts_manager.cc:194] Registered new tserver with Master: 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761)
I20240502 14:06:04.680943 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50254
I20240502 14:06:05.210760 23189 consensus_queue.cc:237] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:05.217523 23343 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.219264 23817 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:05.220360 23268 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.240624 23817 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:06:05.249761 23161 consensus_peers.cc:485] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b0ebc495868467d95f7e6b636732483. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:05.249708 23965 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.251844 23344 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.272413 23092 catalog_manager.cc:5484] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:05.274631 23268 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.278384 23344 consensus_queue.cc:237] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:05.289685 23189 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.291991 23525 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:05.295439 23268 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.297031 23947 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:05.313879 23525 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.315168 23268 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.330773 23196 consensus_queue.cc:237] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
W20240502 14:06:05.332691 23312 consensus_peers.cc:485] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12b5382d476949269a01d25f2e60c2d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:05.333902 23189 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.334367 23089 catalog_manager.cc:5484] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:05.348654 23344 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:05.348825 23161 consensus_peers.cc:485] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b90f20f57cf34057a5e3301a044ad3ca. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:05.349800 23965 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:05.350757 23268 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.352241 23965 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:05.358038 23817 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.362011 23268 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.362190 23344 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.380015 23091 catalog_manager.cc:5484] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.406246 23265 consensus_queue.cc:237] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:05.416221 23344 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.417860 23196 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:05.418010 23958 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:05.420439 23958 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:06:05.432202 23237 consensus_peers.cc:485] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c123d564235427795f0271d9ae4213c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:05.432595 23729 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.437623 23339 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.441792 23196 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:05.449913 23089 catalog_manager.cc:5484] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:05.683660 23954 heartbeater.cc:503] Master 127.21.129.62:37293 was elected leader, sending a full tablet report...
I20240502 14:06:05.701494 23988 ts_tablet_manager.cc:902] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:06:05.709908 23988 tablet_copy_client.cc:308] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.1:38171
I20240502 14:06:05.729554 23207 tablet_copy_service.cc:140] P cc4007c6e59b4361a155c5e8a79bbfb7: Received BeginTabletCopySession request for tablet 6b0ebc495868467d95f7e6b636732483 from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:33374)
I20240502 14:06:05.730170 23207 tablet_copy_service.cc:161] P cc4007c6e59b4361a155c5e8a79bbfb7: Beginning new tablet copy session on tablet 6b0ebc495868467d95f7e6b636732483 from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:33374: session id = 67a108c8df8d451599ff168ac618d878-6b0ebc495868467d95f7e6b636732483
I20240502 14:06:05.737998 23207 tablet_copy_source_session.cc:205] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:05.741528 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ebc495868467d95f7e6b636732483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:05.754411 23988 tablet_copy_client.cc:783] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:05.754985 23988 tablet_copy_client.cc:647] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:05.759229 23988 tablet_copy_client.cc:523] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:05.767145 23988 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:05.786489 23988 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:05.787209 23988 tablet_bootstrap.cc:492] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:05.787788 23988 ts_tablet_manager.cc:1367] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.011s
I20240502 14:06:05.789995 23988 raft_consensus.cc:348] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:05.790750 23988 raft_consensus.cc:729] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:05.791743 23988 consensus_queue.cc:260] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:05.796664 23988 ts_tablet_manager.cc:1398] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20240502 14:06:05.798251 23207 tablet_copy_service.cc:342] P cc4007c6e59b4361a155c5e8a79bbfb7: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-6b0ebc495868467d95f7e6b636732483 received from {username='slave'} at 127.0.0.1:33374
I20240502 14:06:05.798601 23207 tablet_copy_service.cc:433] P cc4007c6e59b4361a155c5e8a79bbfb7: ending tablet copy session 67a108c8df8d451599ff168ac618d878-6b0ebc495868467d95f7e6b636732483 on tablet 6b0ebc495868467d95f7e6b636732483 with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:05.878235 23988 ts_tablet_manager.cc:902] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:05.900665 24000 ts_tablet_manager.cc:902] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:05.906961 24001 ts_tablet_manager.cc:902] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:06:05.940174 24000 tablet_copy_client.cc:308] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.2:34169
I20240502 14:06:05.940644 23988 tablet_copy_client.cc:308] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.3:45179
I20240502 14:06:05.953636 23283 tablet_copy_service.cc:140] P 82490d85cd3d4a1180dc3ecabdd88598: Received BeginTabletCopySession request for tablet 6c123d564235427795f0271d9ae4213c from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:56794)
I20240502 14:06:05.954167 23283 tablet_copy_service.cc:161] P 82490d85cd3d4a1180dc3ecabdd88598: Beginning new tablet copy session on tablet 6c123d564235427795f0271d9ae4213c from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:56794: session id = 67a108c8df8d451599ff168ac618d878-6c123d564235427795f0271d9ae4213c
I20240502 14:06:05.962139 23283 tablet_copy_source_session.cc:205] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:05.965154 24000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c123d564235427795f0271d9ae4213c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:05.983387 24000 tablet_copy_client.cc:783] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:05.983882 24000 tablet_copy_client.cc:647] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:05.987998 24000 tablet_copy_client.cc:523] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:05.991492 23357 tablet_copy_service.cc:140] P 980e09c2959e408db93d72b7125ccf63: Received BeginTabletCopySession request for tablet 12b5382d476949269a01d25f2e60c2d8 from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:57164)
I20240502 14:06:05.992117 23357 tablet_copy_service.cc:161] P 980e09c2959e408db93d72b7125ccf63: Beginning new tablet copy session on tablet 12b5382d476949269a01d25f2e60c2d8 from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:57164: session id = 67a108c8df8d451599ff168ac618d878-12b5382d476949269a01d25f2e60c2d8
I20240502 14:06:05.996420 24000 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:06.000540 23357 tablet_copy_source_session.cc:205] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:06.003968 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b5382d476949269a01d25f2e60c2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:06.005257 24001 tablet_copy_client.cc:308] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.1:38171
I20240502 14:06:06.016680 23207 tablet_copy_service.cc:140] P cc4007c6e59b4361a155c5e8a79bbfb7: Received BeginTabletCopySession request for tablet b90f20f57cf34057a5e3301a044ad3ca from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:33374)
I20240502 14:06:06.017215 23207 tablet_copy_service.cc:161] P cc4007c6e59b4361a155c5e8a79bbfb7: Beginning new tablet copy session on tablet b90f20f57cf34057a5e3301a044ad3ca from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:33374: session id = 67a108c8df8d451599ff168ac618d878-b90f20f57cf34057a5e3301a044ad3ca
I20240502 14:06:06.017369 23988 tablet_copy_client.cc:783] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:06.017884 23988 tablet_copy_client.cc:647] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:06.022899 23988 tablet_copy_client.cc:523] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:06.033519 23988 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:06.038913 23207 tablet_copy_source_session.cc:205] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:06.041812 24001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90f20f57cf34057a5e3301a044ad3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:06.058569 24000 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:06.059485 24000 tablet_bootstrap.cc:492] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:06.060587 24000 ts_tablet_manager.cc:1367] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.064s user 0.033s sys 0.017s
I20240502 14:06:06.063879 24000 raft_consensus.cc:348] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:06.064800 24000 raft_consensus.cc:729] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:06.065405 24000 consensus_queue.cc:260] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:06.067559 24000 ts_tablet_manager.cc:1398] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20240502 14:06:06.068792 23988 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:06.069437 23988 tablet_bootstrap.cc:492] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:06.069891 23283 tablet_copy_service.cc:342] P 82490d85cd3d4a1180dc3ecabdd88598: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-6c123d564235427795f0271d9ae4213c received from {username='slave'} at 127.0.0.1:56794
I20240502 14:06:06.070093 23988 ts_tablet_manager.cc:1367] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.000s
I20240502 14:06:06.071267 23283 tablet_copy_service.cc:433] P 82490d85cd3d4a1180dc3ecabdd88598: ending tablet copy session 67a108c8df8d451599ff168ac618d878-6c123d564235427795f0271d9ae4213c on tablet 6c123d564235427795f0271d9ae4213c with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:06.073859 24001 tablet_copy_client.cc:783] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:06.077917 23988 raft_consensus.cc:348] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:06.078734 23988 raft_consensus.cc:729] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:06.079352 23988 consensus_queue.cc:260] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:06.080801 24001 tablet_copy_client.cc:647] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:06.082420 23988 ts_tablet_manager.cc:1398] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20240502 14:06:06.084182 24001 tablet_copy_client.cc:523] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:06.084689 23357 tablet_copy_service.cc:342] P 980e09c2959e408db93d72b7125ccf63: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-12b5382d476949269a01d25f2e60c2d8 received from {username='slave'} at 127.0.0.1:57164
I20240502 14:06:06.085086 23357 tablet_copy_service.cc:433] P 980e09c2959e408db93d72b7125ccf63: ending tablet copy session 67a108c8df8d451599ff168ac618d878-12b5382d476949269a01d25f2e60c2d8 on tablet 12b5382d476949269a01d25f2e60c2d8 with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:06.090518 24001 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:06.113375 24001 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:06.114137 24001 tablet_bootstrap.cc:492] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:06.114661 24001 ts_tablet_manager.cc:1367] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.002s
I20240502 14:06:06.117103 24001 raft_consensus.cc:348] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:06.117852 24001 raft_consensus.cc:729] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:06.118427 24001 consensus_queue.cc:260] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:06.121389 24001 ts_tablet_manager.cc:1398] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
I20240502 14:06:06.123292 23207 tablet_copy_service.cc:342] P cc4007c6e59b4361a155c5e8a79bbfb7: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-b90f20f57cf34057a5e3301a044ad3ca received from {username='slave'} at 127.0.0.1:33374
I20240502 14:06:06.123684 23207 tablet_copy_service.cc:433] P cc4007c6e59b4361a155c5e8a79bbfb7: ending tablet copy session 67a108c8df8d451599ff168ac618d878-b90f20f57cf34057a5e3301a044ad3ca on tablet b90f20f57cf34057a5e3301a044ad3ca with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:06.256686 23927 raft_consensus.cc:1203] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:06.290261 23927 raft_consensus.cc:1203] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:06.334618 23927 raft_consensus.cc:1203] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:06.374722 23927 raft_consensus.cc:1203] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:06.743739 23974 raft_consensus.cc:1050] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:06.746232 23974 consensus_queue.cc:237] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:06.753196 23927 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.754812 23268 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.754917 23947 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.757241 23947 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.760710 23189 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.763219 23947 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.774201 23817 raft_consensus.cc:1050] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:06.776296 23817 consensus_queue.cc:237] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:06.780064 23974 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.781248 23927 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.794231 23189 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.795145 23089 catalog_manager.cc:5484] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.802217 23268 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.804023 24019 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.805089 23343 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.806054 23927 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.808292 24019 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.807927 23265 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.811753 23981 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.853350 23981 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.854882 23343 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.870177 23268 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.885066 23928 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.908974 23343 consensus_queue.cc:237] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:06.913055 23090 catalog_manager.cc:5484] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.920243 23265 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:06.923702 23928 raft_consensus.cc:1261] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:06.925796 23947 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:06.926803 23997 consensus_queue.cc:1035] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:06.933818 24025 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.939342 23927 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.939848 23265 raft_consensus.cc:2932] T 12b5382d476949269a01d25f2e60c2d8 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.945050 23189 consensus_queue.cc:237] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:06.953685 24015 raft_consensus.cc:1050] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:06.956024 24015 consensus_queue.cc:237] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:06.957171 23091 catalog_manager.cc:5484] T 12b5382d476949269a01d25f2e60c2d8 P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:06.958038 23344 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:06.959772 23927 raft_consensus.cc:1261] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:06.960914 24026 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:06.963142 24032 consensus_queue.cc:1035] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:06.966261 24032 raft_consensus.cc:1050] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:06.968698 24032 consensus_queue.cc:237] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:06.972225 23926 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.972467 23189 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.976609 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet 12b5382d476949269a01d25f2e60c2d8 with delete_type TABLET_DATA_TOMBSTONED (TS cc4007c6e59b4361a155c5e8a79bbfb7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40056
I20240502 14:06:06.983655 23927 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.984690 23341 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.985011 23817 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.988381 24026 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:06.974642 23985 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240502 14:06:06.991232 24037 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 14:06:06.993855 23344 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:06.997705 24033 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.001255 23339 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.010300 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b0ebc495868467d95f7e6b636732483 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:07.015918 23985 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:07.021999 24040 tablet_replica.cc:330] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: stopping tablet replica
I20240502 14:06:07.023332 23928 raft_consensus.cc:2932] T 6b0ebc495868467d95f7e6b636732483 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.026362 23091 catalog_manager.cc:5484] T 6b0ebc495868467d95f7e6b636732483 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.041728 23265 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:07.044355 24033 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.047659 23817 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:07.056288 23341 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.060856 23928 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.067392 23256 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0ebc495868467d95f7e6b636732483 with delete_type TABLET_DATA_TOMBSTONED (TS 82490d85cd3d4a1180dc3ecabdd88598 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52056
I20240502 14:06:07.077301 24040 raft_consensus.cc:2229] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:07.078043 24040 raft_consensus.cc:2258] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:07.081179 24040 ts_tablet_manager.cc:1875] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:07.084007 24041 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.086942 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12b5382d476949269a01d25f2e60c2d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:07.093271 23344 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.104497 24040 ts_tablet_manager.cc:1888] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:07.104933 24040 log.cc:1196] T 12b5382d476949269a01d25f2e60c2d8 P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root/wals/12b5382d476949269a01d25f2e60c2d8
I20240502 14:06:07.106446 23079 catalog_manager.cc:4830] TS cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): tablet 12b5382d476949269a01d25f2e60c2d8 (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:07.112010 23092 catalog_manager.cc:5484] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:07.115672 23927 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.121371 23196 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.126052 23089 catalog_manager.cc:5484] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:07.155119 24047 tablet_replica.cc:330] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: stopping tablet replica
I20240502 14:06:07.172930 24047 raft_consensus.cc:2229] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:07.173625 24047 raft_consensus.cc:2258] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:07.176472 24047 ts_tablet_manager.cc:1875] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:07.172595 23266 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.192158 24047 ts_tablet_manager.cc:1888] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:07.192559 24047 log.cc:1196] T 6b0ebc495868467d95f7e6b636732483 P 82490d85cd3d4a1180dc3ecabdd88598: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root/wals/6b0ebc495868467d95f7e6b636732483
I20240502 14:06:07.194129 23076 catalog_manager.cc:4830] TS 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): tablet 6b0ebc495868467d95f7e6b636732483 (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:07.199539 23265 consensus_queue.cc:237] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:07.203487 23196 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:07.204308 23928 raft_consensus.cc:1261] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:07.204594 24041 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:07.206738 24039 consensus_queue.cc:1035] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:07.216202 24041 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.218979 23191 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.219245 23928 raft_consensus.cc:2932] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.233443 23092 catalog_manager.cc:5484] T 6c123d564235427795f0271d9ae4213c P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.234799 23076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c123d564235427795f0271d9ae4213c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:07.248780 23334 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c123d564235427795f0271d9ae4213c with delete_type TABLET_DATA_TOMBSTONED (TS 980e09c2959e408db93d72b7125ccf63 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56144
I20240502 14:06:07.252599 24050 tablet_replica.cc:330] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: stopping tablet replica
I20240502 14:06:07.253633 24050 raft_consensus.cc:2229] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:07.254268 24050 raft_consensus.cc:2258] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:07.257010 24050 ts_tablet_manager.cc:1875] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:07.268370 24050 ts_tablet_manager.cc:1888] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:07.268661 24050 log.cc:1196] T 6c123d564235427795f0271d9ae4213c P 980e09c2959e408db93d72b7125ccf63: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root/wals/6c123d564235427795f0271d9ae4213c
I20240502 14:06:07.269912 23079 catalog_manager.cc:4830] TS 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): tablet 6c123d564235427795f0271d9ae4213c (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:07.544665 23189 consensus_queue.cc:237] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:07.548229 23266 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:07.548873 23928 raft_consensus.cc:1261] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:07.549676 24033 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:07.550544 24026 consensus_queue.cc:1035] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:07.557920 23981 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.561780 23266 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.564692 23928 raft_consensus.cc:2932] T b90f20f57cf34057a5e3301a044ad3ca P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:07.572974 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b90f20f57cf34057a5e3301a044ad3ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:07.575125 23091 catalog_manager.cc:5484] T b90f20f57cf34057a5e3301a044ad3ca P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:07.584997 23334 tablet_service.cc:1514] Processing DeleteTablet for tablet b90f20f57cf34057a5e3301a044ad3ca with delete_type TABLET_DATA_TOMBSTONED (TS 980e09c2959e408db93d72b7125ccf63 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56144
I20240502 14:06:07.585889 24050 tablet_replica.cc:330] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: stopping tablet replica
I20240502 14:06:07.586608 24050 raft_consensus.cc:2229] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:07.587108 24050 raft_consensus.cc:2258] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:07.590449 24050 ts_tablet_manager.cc:1875] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:07.609737 24050 ts_tablet_manager.cc:1888] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:07.610082 24050 log.cc:1196] T b90f20f57cf34057a5e3301a044ad3ca P 980e09c2959e408db93d72b7125ccf63: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root/wals/b90f20f57cf34057a5e3301a044ad3ca
I20240502 14:06:07.611388 23079 catalog_manager.cc:4830] TS 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): tablet b90f20f57cf34057a5e3301a044ad3ca (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:08.643417 23341 consensus_queue.cc:237] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:08.649303 23268 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:08.650213 23312 consensus_peers.cc:485] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cccf9e842a284cf8bcc8d60d0dd1a22b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:08.650280 23189 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:08.650574 24025 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:08.651731 23974 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.658617 24025 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.660471 23268 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.661612 23189 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.669721 23266 consensus_queue.cc:237] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:08.677603 23092 catalog_manager.cc:5484] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.690697 23189 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:08.691247 23237 consensus_peers.cc:485] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afca5b05bac0423ca02254a903b07eca. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:08.691702 23344 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:08.693029 24042 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.694221 24058 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.713378 24039 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.716210 23189 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.716817 23344 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.744320 23189 consensus_queue.cc:237] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:08.745824 23091 catalog_manager.cc:5484] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.765177 23344 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:08.765687 23161 consensus_peers.cc:485] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb3ff7f6ffd34055abb266d2b25e2e4d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:08.766865 24026 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.771991 23266 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:08.773319 23981 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.775977 23981 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.776995 23341 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.791679 23341 consensus_queue.cc:237] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:08.793681 23089 catalog_manager.cc:5484] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.801913 23268 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:08.801272 23266 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.802546 23189 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:08.803578 23312 consensus_peers.cc:485] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f6c31e65498481696374e87a81890fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:08.803948 24025 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.806846 24025 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:08.823922 24055 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.828707 23268 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.828791 23191 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:08.845484 23089 catalog_manager.cc:5484] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:09.100397 24071 ts_tablet_manager.cc:902] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:09.105116 24071 tablet_copy_client.cc:308] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.2:34169
I20240502 14:06:09.106523 23283 tablet_copy_service.cc:140] P 82490d85cd3d4a1180dc3ecabdd88598: Received BeginTabletCopySession request for tablet afca5b05bac0423ca02254a903b07eca from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:56794)
I20240502 14:06:09.107019 23283 tablet_copy_service.cc:161] P 82490d85cd3d4a1180dc3ecabdd88598: Beginning new tablet copy session on tablet afca5b05bac0423ca02254a903b07eca from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:56794: session id = 67a108c8df8d451599ff168ac618d878-afca5b05bac0423ca02254a903b07eca
I20240502 14:06:09.114307 23283 tablet_copy_source_session.cc:205] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:09.117007 24071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afca5b05bac0423ca02254a903b07eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:09.129694 24071 tablet_copy_client.cc:783] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:09.130180 24071 tablet_copy_client.cc:647] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:09.133525 24071 tablet_copy_client.cc:523] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:09.139685 24071 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:09.156121 24071 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:09.156881 24071 tablet_bootstrap.cc:492] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:09.157292 24077 ts_tablet_manager.cc:902] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:09.157471 24071 ts_tablet_manager.cc:1367] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240502 14:06:09.160243 24071 raft_consensus.cc:348] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.161053 24071 raft_consensus.cc:729] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:09.161638 24071 consensus_queue.cc:260] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.169337 24071 ts_tablet_manager.cc:1398] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20240502 14:06:09.170704 23283 tablet_copy_service.cc:342] P 82490d85cd3d4a1180dc3ecabdd88598: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-afca5b05bac0423ca02254a903b07eca received from {username='slave'} at 127.0.0.1:56794
I20240502 14:06:09.171079 23283 tablet_copy_service.cc:433] P 82490d85cd3d4a1180dc3ecabdd88598: ending tablet copy session 67a108c8df8d451599ff168ac618d878-afca5b05bac0423ca02254a903b07eca on tablet afca5b05bac0423ca02254a903b07eca with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:09.176306 24077 tablet_copy_client.cc:308] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.3:45179
I20240502 14:06:09.178071 23357 tablet_copy_service.cc:140] P 980e09c2959e408db93d72b7125ccf63: Received BeginTabletCopySession request for tablet cccf9e842a284cf8bcc8d60d0dd1a22b from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:57164)
I20240502 14:06:09.178578 23357 tablet_copy_service.cc:161] P 980e09c2959e408db93d72b7125ccf63: Beginning new tablet copy session on tablet cccf9e842a284cf8bcc8d60d0dd1a22b from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:57164: session id = 67a108c8df8d451599ff168ac618d878-cccf9e842a284cf8bcc8d60d0dd1a22b
I20240502 14:06:09.185920 23357 tablet_copy_source_session.cc:205] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:09.190465 24077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf9e842a284cf8bcc8d60d0dd1a22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:09.205309 24077 tablet_copy_client.cc:783] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:09.205826 24077 tablet_copy_client.cc:647] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:09.210204 24077 tablet_copy_client.cc:523] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:09.216719 24077 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:09.221933 24071 ts_tablet_manager.cc:902] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:06:09.233528 24077 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:09.234290 24077 tablet_bootstrap.cc:492] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:09.234900 24077 ts_tablet_manager.cc:1367] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20240502 14:06:09.237602 24077 raft_consensus.cc:348] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.238449 24077 raft_consensus.cc:729] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:09.239038 24077 consensus_queue.cc:260] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.241030 24077 ts_tablet_manager.cc:1398] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20240502 14:06:09.242275 24071 tablet_copy_client.cc:308] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.1:38171
I20240502 14:06:09.243827 23357 tablet_copy_service.cc:342] P 980e09c2959e408db93d72b7125ccf63: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-cccf9e842a284cf8bcc8d60d0dd1a22b received from {username='slave'} at 127.0.0.1:57164
I20240502 14:06:09.244113 23207 tablet_copy_service.cc:140] P cc4007c6e59b4361a155c5e8a79bbfb7: Received BeginTabletCopySession request for tablet eb3ff7f6ffd34055abb266d2b25e2e4d from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:33374)
I20240502 14:06:09.244665 23207 tablet_copy_service.cc:161] P cc4007c6e59b4361a155c5e8a79bbfb7: Beginning new tablet copy session on tablet eb3ff7f6ffd34055abb266d2b25e2e4d from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:33374: session id = 67a108c8df8d451599ff168ac618d878-eb3ff7f6ffd34055abb266d2b25e2e4d
I20240502 14:06:09.244216 23357 tablet_copy_service.cc:433] P 980e09c2959e408db93d72b7125ccf63: ending tablet copy session 67a108c8df8d451599ff168ac618d878-cccf9e842a284cf8bcc8d60d0dd1a22b on tablet cccf9e842a284cf8bcc8d60d0dd1a22b with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:09.253087 23207 tablet_copy_source_session.cc:205] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:09.255759 24071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ff7f6ffd34055abb266d2b25e2e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:09.267697 24071 tablet_copy_client.cc:783] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:09.268445 24071 tablet_copy_client.cc:647] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:09.271984 24071 tablet_copy_client.cc:523] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:09.282374 24071 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:09.298548 24071 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:09.299316 24071 tablet_bootstrap.cc:492] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:09.299778 24071 ts_tablet_manager.cc:1367] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240502 14:06:09.301950 24071 raft_consensus.cc:348] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.302534 24071 raft_consensus.cc:729] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:09.303097 24071 consensus_queue.cc:260] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.305191 24071 ts_tablet_manager.cc:1398] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:06:09.307018 23207 tablet_copy_service.cc:342] P cc4007c6e59b4361a155c5e8a79bbfb7: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-eb3ff7f6ffd34055abb266d2b25e2e4d received from {username='slave'} at 127.0.0.1:33374
I20240502 14:06:09.307350 23207 tablet_copy_service.cc:433] P cc4007c6e59b4361a155c5e8a79bbfb7: ending tablet copy session 67a108c8df8d451599ff168ac618d878-eb3ff7f6ffd34055abb266d2b25e2e4d on tablet eb3ff7f6ffd34055abb266d2b25e2e4d with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:09.322175 24071 ts_tablet_manager.cc:902] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:09.326213 24071 tablet_copy_client.cc:308] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.3:45179
I20240502 14:06:09.327509 23357 tablet_copy_service.cc:140] P 980e09c2959e408db93d72b7125ccf63: Received BeginTabletCopySession request for tablet 5f6c31e65498481696374e87a81890fa from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:57164)
I20240502 14:06:09.327967 23357 tablet_copy_service.cc:161] P 980e09c2959e408db93d72b7125ccf63: Beginning new tablet copy session on tablet 5f6c31e65498481696374e87a81890fa from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:57164: session id = 67a108c8df8d451599ff168ac618d878-5f6c31e65498481696374e87a81890fa
I20240502 14:06:09.333989 23357 tablet_copy_source_session.cc:205] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:09.336249 24071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c31e65498481696374e87a81890fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:09.348122 24071 tablet_copy_client.cc:783] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:09.348598 24071 tablet_copy_client.cc:647] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:09.352609 24071 tablet_copy_client.cc:523] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:09.358832 24071 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:09.375116 24071 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:09.375878 24071 tablet_bootstrap.cc:492] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:09.376444 24071 ts_tablet_manager.cc:1367] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240502 14:06:09.378656 24071 raft_consensus.cc:348] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.379455 24071 raft_consensus.cc:729] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:09.379972 24071 consensus_queue.cc:260] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:09.382211 24071 ts_tablet_manager.cc:1398] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:06:09.385852 23357 tablet_copy_service.cc:342] P 980e09c2959e408db93d72b7125ccf63: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-5f6c31e65498481696374e87a81890fa received from {username='slave'} at 127.0.0.1:57164
I20240502 14:06:09.386168 23357 tablet_copy_service.cc:433] P 980e09c2959e408db93d72b7125ccf63: ending tablet copy session 67a108c8df8d451599ff168ac618d878-5f6c31e65498481696374e87a81890fa on tablet 5f6c31e65498481696374e87a81890fa with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:09.556721 23928 raft_consensus.cc:1203] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:09.725467 23928 raft_consensus.cc:1203] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:09.928308 23928 raft_consensus.cc:1203] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:10.027783 23926 raft_consensus.cc:1203] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:10.028767 24094 raft_consensus.cc:1050] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:10.031273 24094 consensus_queue.cc:237] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.046566 23926 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.048354 23270 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.048872 24025 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.049721 24094 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.070796 23197 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.074982 24094 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.081143 24025 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.095942 23926 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.099481 23197 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.105044 24064 raft_consensus.cc:1050] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:10.107281 24064 consensus_queue.cc:237] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.109575 23092 catalog_manager.cc:5484] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:10.120157 23269 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.130177 23340 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.131321 24042 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.134943 23926 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.135970 23197 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.139216 24098 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.140854 24096 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.152494 24096 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.157032 23926 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.158686 23340 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.168572 23197 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.185971 23090 catalog_manager.cc:5484] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.191231 23339 consensus_queue.cc:237] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.198570 23926 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.198714 23197 raft_consensus.cc:1261] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.200018 24094 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.200843 24102 consensus_queue.cc:1035] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.209646 24102 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.213610 23926 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.221796 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cccf9e842a284cf8bcc8d60d0dd1a22b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:10.219949 23197 raft_consensus.cc:2932] T cccf9e842a284cf8bcc8d60d0dd1a22b P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.226904 23090 catalog_manager.cc:5484] T cccf9e842a284cf8bcc8d60d0dd1a22b P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.229328 23269 consensus_queue.cc:237] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.236701 23926 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.237995 23340 raft_consensus.cc:1261] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.239372 24097 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:10.242460 23256 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf9e842a284cf8bcc8d60d0dd1a22b with delete_type TABLET_DATA_TOMBSTONED (TS 82490d85cd3d4a1180dc3ecabdd88598 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52056
I20240502 14:06:10.247763 24064 consensus_queue.cc:1035] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.257716 24116 tablet_replica.cc:330] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: stopping tablet replica
I20240502 14:06:10.258002 24096 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.260468 24116 raft_consensus.cc:2229] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:10.261157 24116 raft_consensus.cc:2258] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:10.261410 23926 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.267644 23076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afca5b05bac0423ca02254a903b07eca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:10.269847 24116 ts_tablet_manager.cc:1875] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:10.269116 23340 raft_consensus.cc:2932] T afca5b05bac0423ca02254a903b07eca P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.281615 23092 catalog_manager.cc:5484] T afca5b05bac0423ca02254a903b07eca P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:10.286018 24116 ts_tablet_manager.cc:1888] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:10.286459 24116 log.cc:1196] T cccf9e842a284cf8bcc8d60d0dd1a22b P 82490d85cd3d4a1180dc3ecabdd88598: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root/wals/cccf9e842a284cf8bcc8d60d0dd1a22b
I20240502 14:06:10.287917 23076 catalog_manager.cc:4830] TS 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): tablet cccf9e842a284cf8bcc8d60d0dd1a22b (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:10.297207 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet afca5b05bac0423ca02254a903b07eca with delete_type TABLET_DATA_TOMBSTONED (TS cc4007c6e59b4361a155c5e8a79bbfb7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40056
I20240502 14:06:10.301784 24117 tablet_replica.cc:330] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: stopping tablet replica
I20240502 14:06:10.302839 24117 raft_consensus.cc:2229] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:10.304507 24051 raft_consensus.cc:1050] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:10.304903 24117 raft_consensus.cc:2258] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:10.306977 24051 consensus_queue.cc:237] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.308061 24117 ts_tablet_manager.cc:1875] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:10.314921 23926 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.316557 24101 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.317049 23340 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.318658 24051 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.320586 24117 ts_tablet_manager.cc:1888] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:10.320991 24117 log.cc:1196] T afca5b05bac0423ca02254a903b07eca P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root/wals/afca5b05bac0423ca02254a903b07eca
I20240502 14:06:10.323045 23079 catalog_manager.cc:4830] TS cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): tablet afca5b05bac0423ca02254a903b07eca (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:10.326722 23270 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.328301 24095 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.329525 24101 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.330880 23339 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.339928 23926 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.340875 23270 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.346799 23092 catalog_manager.cc:5484] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:10.449656 24102 raft_consensus.cc:1050] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:10.452065 24102 consensus_queue.cc:237] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.459645 23270 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.459842 23926 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.461133 24025 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.461894 24103 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.462424 23197 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:10.465355 24025 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:10.477629 24103 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.479332 23270 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.484603 23191 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.490693 23926 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.517503 23091 catalog_manager.cc:5484] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.540349 23340 consensus_queue.cc:237] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.548794 23270 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.549861 24025 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.552677 23926 raft_consensus.cc:1261] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.553957 24025 consensus_queue.cc:1035] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.561340 24094 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.569015 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f6c31e65498481696374e87a81890fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:10.575862 23926 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.576884 23092 catalog_manager.cc:5484] T 5f6c31e65498481696374e87a81890fa P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:10.571807 23270 raft_consensus.cc:2932] T 5f6c31e65498481696374e87a81890fa P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.592326 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c31e65498481696374e87a81890fa with delete_type TABLET_DATA_TOMBSTONED (TS cc4007c6e59b4361a155c5e8a79bbfb7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40056
I20240502 14:06:10.593883 24117 tablet_replica.cc:330] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: stopping tablet replica
I20240502 14:06:10.594882 24117 raft_consensus.cc:2229] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:10.600500 24117 raft_consensus.cc:2258] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:10.609019 24117 ts_tablet_manager.cc:1875] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:10.621155 24117 ts_tablet_manager.cc:1888] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:10.621506 24117 log.cc:1196] T 5f6c31e65498481696374e87a81890fa P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root/wals/5f6c31e65498481696374e87a81890fa
I20240502 14:06:10.622728 23079 catalog_manager.cc:4830] TS cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): tablet 5f6c31e65498481696374e87a81890fa (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:10.724056 23197 consensus_queue.cc:237] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:10.728543 23926 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.728652 23270 raft_consensus.cc:1261] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:10.730165 24095 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.731027 24051 consensus_queue.cc:1035] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:10.737933 24026 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.738821 23269 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.739470 23926 raft_consensus.cc:2932] T eb3ff7f6ffd34055abb266d2b25e2e4d P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.754405 23092 catalog_manager.cc:5484] T eb3ff7f6ffd34055abb266d2b25e2e4d P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:10.752242 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb3ff7f6ffd34055abb266d2b25e2e4d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:10.765594 23334 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3ff7f6ffd34055abb266d2b25e2e4d with delete_type TABLET_DATA_TOMBSTONED (TS 980e09c2959e408db93d72b7125ccf63 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56144
I20240502 14:06:10.768185 24126 tablet_replica.cc:330] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: stopping tablet replica
I20240502 14:06:10.769055 24126 raft_consensus.cc:2229] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:10.770001 24126 raft_consensus.cc:2258] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:10.772940 24126 ts_tablet_manager.cc:1875] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:10.783528 24126 ts_tablet_manager.cc:1888] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:10.783877 24126 log.cc:1196] T eb3ff7f6ffd34055abb266d2b25e2e4d P 980e09c2959e408db93d72b7125ccf63: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root/wals/eb3ff7f6ffd34055abb266d2b25e2e4d
I20240502 14:06:10.785264 23079 catalog_manager.cc:4830] TS 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): tablet eb3ff7f6ffd34055abb266d2b25e2e4d (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
W20240502 14:06:10.933717 23368 debug-util.cc:398] Leaking SignalData structure 0x7b08001b5da0 after lost signal to thread 22023
W20240502 14:06:10.934547 23368 debug-util.cc:398] Leaking SignalData structure 0x7b08001df8e0 after lost signal to thread 23128
W20240502 14:06:10.935464 23368 debug-util.cc:398] Leaking SignalData structure 0x7b080031b5e0 after lost signal to thread 23221
W20240502 14:06:10.936342 23368 debug-util.cc:398] Leaking SignalData structure 0x7b08002a3b00 after lost signal to thread 23297
W20240502 14:06:10.937285 23368 debug-util.cc:398] Leaking SignalData structure 0x7b080026cd20 after lost signal to thread 23371
W20240502 14:06:10.938351 23368 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1460 after lost signal to thread 23953
I20240502 14:06:11.832669 23340 consensus_queue.cc:237] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:11.839442 23269 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:11.840162 23312 consensus_peers.cc:485] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b400e3ddeee74798b411ee7a4e285b7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:11.840926 24094 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:11.843852 23197 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:11.846055 24094 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:11.849642 24103 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.852559 23269 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.867228 23197 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.873133 23191 consensus_queue.cc:237] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:11.875030 23091 catalog_manager.cc:5484] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.903563 23269 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:11.903743 23161 consensus_peers.cc:485] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6132f4449de74e15a3cbe93f03a74b71. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:11.908905 23340 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:11.909471 24101 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:11.910944 24127 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:11.932471 24101 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.934293 23344 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.936129 23270 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.947578 23191 consensus_queue.cc:237] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
W20240502 14:06:11.961843 23161 consensus_peers.cc:485] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c5069e564364a3b82e55d410fc03cec. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:11.962214 23344 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:11.964483 23269 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:11.967938 24101 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:11.969588 24101 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:11.982292 24101 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.984599 23344 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.988190 23269 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:11.996642 23270 consensus_queue.cc:237] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.008742 23090 catalog_manager.cc:5484] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:12.015854 23344 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:12.016301 23191 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:12.017277 23237 consensus_peers.cc:485] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b17aa98c3f04a58a5613d8b4af43f2e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:12.019042 24098 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:12.020738 24098 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:12.028445 23092 catalog_manager.cc:5484] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:12.042004 24140 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:12.052435 23191 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:12.062906 23344 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:12.065241 23090 catalog_manager.cc:5484] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:12.298544 24149 ts_tablet_manager.cc:902] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:06:12.323364 24149 tablet_copy_client.cc:308] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.1:38171
I20240502 14:06:12.325162 23207 tablet_copy_service.cc:140] P cc4007c6e59b4361a155c5e8a79bbfb7: Received BeginTabletCopySession request for tablet 6132f4449de74e15a3cbe93f03a74b71 from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:33374)
I20240502 14:06:12.325744 23207 tablet_copy_service.cc:161] P cc4007c6e59b4361a155c5e8a79bbfb7: Beginning new tablet copy session on tablet 6132f4449de74e15a3cbe93f03a74b71 from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:33374: session id = 67a108c8df8d451599ff168ac618d878-6132f4449de74e15a3cbe93f03a74b71
I20240502 14:06:12.333679 23207 tablet_copy_source_session.cc:205] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:12.336575 24149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6132f4449de74e15a3cbe93f03a74b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:12.349009 24149 tablet_copy_client.cc:783] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:12.349531 24149 tablet_copy_client.cc:647] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:12.357230 24149 tablet_copy_client.cc:523] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:12.369778 24149 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:12.387681 24149 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:12.388298 24149 tablet_bootstrap.cc:492] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:12.388911 24149 ts_tablet_manager.cc:1367] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240502 14:06:12.390980 24149 raft_consensus.cc:348] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.391809 24149 raft_consensus.cc:729] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:12.392397 24149 consensus_queue.cc:260] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.395537 24149 ts_tablet_manager.cc:1398] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 14:06:12.397346 23207 tablet_copy_service.cc:342] P cc4007c6e59b4361a155c5e8a79bbfb7: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-6132f4449de74e15a3cbe93f03a74b71 received from {username='slave'} at 127.0.0.1:33374
I20240502 14:06:12.397656 23207 tablet_copy_service.cc:433] P cc4007c6e59b4361a155c5e8a79bbfb7: ending tablet copy session 67a108c8df8d451599ff168ac618d878-6132f4449de74e15a3cbe93f03a74b71 on tablet 6132f4449de74e15a3cbe93f03a74b71 with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:12.424432 24149 ts_tablet_manager.cc:902] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
I20240502 14:06:12.452060 24158 ts_tablet_manager.cc:902] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
I20240502 14:06:12.470805 24160 ts_tablet_manager.cc:902] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
I20240502 14:06:12.539945 24149 tablet_copy_client.cc:308] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.2:34169
I20240502 14:06:12.541940 23283 tablet_copy_service.cc:140] P 82490d85cd3d4a1180dc3ecabdd88598: Received BeginTabletCopySession request for tablet 8b17aa98c3f04a58a5613d8b4af43f2e from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:56794)
I20240502 14:06:12.542467 23283 tablet_copy_service.cc:161] P 82490d85cd3d4a1180dc3ecabdd88598: Beginning new tablet copy session on tablet 8b17aa98c3f04a58a5613d8b4af43f2e from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:56794: session id = 67a108c8df8d451599ff168ac618d878-8b17aa98c3f04a58a5613d8b4af43f2e
I20240502 14:06:12.561057 23283 tablet_copy_source_session.cc:205] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:12.564800 24149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b17aa98c3f04a58a5613d8b4af43f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:12.592891 24149 tablet_copy_client.cc:783] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:12.593501 24149 tablet_copy_client.cc:647] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:12.601418 24149 tablet_copy_client.cc:523] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:12.609766 24149 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
W20240502 14:06:12.636962 23238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 45881.5us
I20240502 14:06:12.657012 24149 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:12.657959 24149 tablet_bootstrap.cc:492] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:12.658646 24149 ts_tablet_manager.cc:1367] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.005s
I20240502 14:06:12.661893 24149 raft_consensus.cc:348] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.662809 24149 raft_consensus.cc:729] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:12.663556 24149 consensus_queue.cc:260] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.666251 24149 ts_tablet_manager.cc:1398] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:06:12.668290 23283 tablet_copy_service.cc:342] P 82490d85cd3d4a1180dc3ecabdd88598: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-8b17aa98c3f04a58a5613d8b4af43f2e received from {username='slave'} at 127.0.0.1:56794
I20240502 14:06:12.668678 23283 tablet_copy_service.cc:433] P 82490d85cd3d4a1180dc3ecabdd88598: ending tablet copy session 67a108c8df8d451599ff168ac618d878-8b17aa98c3f04a58a5613d8b4af43f2e on tablet 8b17aa98c3f04a58a5613d8b4af43f2e with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:12.673910 24158 tablet_copy_client.cc:308] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.1:38171
I20240502 14:06:12.675572 23207 tablet_copy_service.cc:140] P cc4007c6e59b4361a155c5e8a79bbfb7: Received BeginTabletCopySession request for tablet 6c5069e564364a3b82e55d410fc03cec from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:33374)
I20240502 14:06:12.676169 24160 tablet_copy_client.cc:308] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: tablet copy: Beginning tablet copy session from remote peer at address 127.21.129.3:45179
I20240502 14:06:12.676086 23207 tablet_copy_service.cc:161] P cc4007c6e59b4361a155c5e8a79bbfb7: Beginning new tablet copy session on tablet 6c5069e564364a3b82e55d410fc03cec from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:33374: session id = 67a108c8df8d451599ff168ac618d878-6c5069e564364a3b82e55d410fc03cec
I20240502 14:06:12.682603 23357 tablet_copy_service.cc:140] P 980e09c2959e408db93d72b7125ccf63: Received BeginTabletCopySession request for tablet b400e3ddeee74798b411ee7a4e285b7a from peer 67a108c8df8d451599ff168ac618d878 ({username='slave'} at 127.0.0.1:57164)
I20240502 14:06:12.683116 23357 tablet_copy_service.cc:161] P 980e09c2959e408db93d72b7125ccf63: Beginning new tablet copy session on tablet b400e3ddeee74798b411ee7a4e285b7a from peer 67a108c8df8d451599ff168ac618d878 at {username='slave'} at 127.0.0.1:57164: session id = 67a108c8df8d451599ff168ac618d878-b400e3ddeee74798b411ee7a4e285b7a
I20240502 14:06:12.684938 23207 tablet_copy_source_session.cc:205] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:12.688596 24158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5069e564364a3b82e55d410fc03cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:12.689998 23357 tablet_copy_source_session.cc:205] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:12.693751 24160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b400e3ddeee74798b411ee7a4e285b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:12.703572 24158 tablet_copy_client.cc:783] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:12.704123 24158 tablet_copy_client.cc:647] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:12.708349 24158 tablet_copy_client.cc:523] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:12.717152 24158 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:12.727329 24160 tablet_copy_client.cc:783] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:12.731827 24160 tablet_copy_client.cc:647] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:12.735699 24160 tablet_copy_client.cc:523] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:12.746766 24160 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: Bootstrap starting.
I20240502 14:06:12.752493 24158 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:12.753330 24158 tablet_bootstrap.cc:492] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:12.753958 24158 ts_tablet_manager.cc:1367] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.013s
I20240502 14:06:12.756722 24158 raft_consensus.cc:348] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.757519 24158 raft_consensus.cc:729] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:12.758127 24158 consensus_queue.cc:260] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.772182 24158 ts_tablet_manager.cc:1398] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240502 14:06:12.774065 23207 tablet_copy_service.cc:342] P cc4007c6e59b4361a155c5e8a79bbfb7: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-6c5069e564364a3b82e55d410fc03cec received from {username='slave'} at 127.0.0.1:33374
I20240502 14:06:12.774442 23207 tablet_copy_service.cc:433] P cc4007c6e59b4361a155c5e8a79bbfb7: ending tablet copy session 67a108c8df8d451599ff168ac618d878-6c5069e564364a3b82e55d410fc03cec on tablet 6c5069e564364a3b82e55d410fc03cec with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:12.797251 24160 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:12.798153 24160 tablet_bootstrap.cc:492] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: Bootstrap complete.
I20240502 14:06:12.799185 24160 ts_tablet_manager.cc:1367] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.008s
I20240502 14:06:12.801888 24160 raft_consensus.cc:348] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.802788 24160 raft_consensus.cc:729] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67a108c8df8d451599ff168ac618d878, State: Initialized, Role: LEARNER
I20240502 14:06:12.803409 24160 consensus_queue.cc:260] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:12.807535 24160 ts_tablet_manager.cc:1398] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:06:12.808813 23357 tablet_copy_service.cc:342] P 980e09c2959e408db93d72b7125ccf63: Request end of tablet copy session 67a108c8df8d451599ff168ac618d878-b400e3ddeee74798b411ee7a4e285b7a received from {username='slave'} at 127.0.0.1:57164
I20240502 14:06:12.809123 23357 tablet_copy_service.cc:433] P 980e09c2959e408db93d72b7125ccf63: ending tablet copy session 67a108c8df8d451599ff168ac618d878-b400e3ddeee74798b411ee7a4e285b7a on tablet b400e3ddeee74798b411ee7a4e285b7a with peer 67a108c8df8d451599ff168ac618d878
I20240502 14:06:12.839857 23926 raft_consensus.cc:1203] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:12.841832 23927 raft_consensus.cc:1203] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:12.964075 23927 raft_consensus.cc:1203] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:13.073554 23927 raft_consensus.cc:1203] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:13.304948 24101 raft_consensus.cc:1050] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:13.307435 24101 consensus_queue.cc:237] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:13.331164 23927 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.331804 23339 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.332967 24127 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:13.333671 23270 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.333787 24101 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:13.343549 24130 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:13.353211 24178 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.354491 23344 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.371707 23927 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.373418 23091 catalog_manager.cc:5484] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:13.392134 23270 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.419409 24064 raft_consensus.cc:1050] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:13.421535 24064 consensus_queue.cc:237] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:13.428217 24130 raft_consensus.cc:1050] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:13.430517 24130 consensus_queue.cc:237] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:13.495594 23927 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.503880 23197 consensus_queue.cc:237] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:13.507587 24127 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:13.514060 23344 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:13.517235 23926 raft_consensus.cc:1261] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:13.522588 24127 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20240502 14:06:13.544361 24101 consensus_queue.cc:1035] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:13.548337 24133 raft_consensus.cc:1050] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63: attempting to promote NON_VOTER 67a108c8df8d451599ff168ac618d878 to VOTER
I20240502 14:06:13.556458 23344 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.560048 24133 consensus_queue.cc:237] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:13.573532 23926 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.575124 24140 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:13.577808 23191 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.601771 23341 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 14:06:13.611426 24186 log.cc:924] Time spent T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20240502 14:06:13.620918 24178 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:13.626124 23269 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.584386 24140 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20240502 14:06:13.629613 23926 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.641000 24130 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.671015 24064 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:13.679378 23927 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 LEARNER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.683243 23196 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.686178 24133 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:13.692479 24192 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:13.707243 23269 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:13.716459 23092 catalog_manager.cc:5484] T 6132f4449de74e15a3cbe93f03a74b71 P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.711422 24133 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
W20240502 14:06:13.804533 23312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.4:46761, user_credentials={real_user=slave}} blocked reactor thread for 81077.5us
W20240502 14:06:13.805047 23162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.3:45179, user_credentials={real_user=slave}} blocked reactor thread for 113600us
I20240502 14:06:13.827881 24101 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
W20240502 14:06:13.833525 23159 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 67045.2us
W20240502 14:06:13.850092 23310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 117569us
I20240502 14:06:13.855135 23928 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.861042 24064 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.867352 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6132f4449de74e15a3cbe93f03a74b71 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:13.874503 23197 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.876834 23266 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
W20240502 14:06:13.896595 23312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.4:46761, user_credentials={real_user=slave}} blocked reactor thread for 47142us
I20240502 14:06:13.902663 23089 catalog_manager.cc:5484] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.913520 23256 tablet_service.cc:1514] Processing DeleteTablet for tablet 6132f4449de74e15a3cbe93f03a74b71 with delete_type TABLET_DATA_TOMBSTONED (TS 82490d85cd3d4a1180dc3ecabdd88598 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52056
I20240502 14:06:13.924122 23344 raft_consensus.cc:2932] T 6132f4449de74e15a3cbe93f03a74b71 P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.934924 23341 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.924206 23928 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.962996 24133 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.971874 23927 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:13.966542 23092 catalog_manager.cc:5484] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:06:13.983302 23310 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.2:34169, user_credentials={real_user=slave}} blocked reactor thread for 131670us
I20240502 14:06:13.985395 24213 tablet_replica.cc:330] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: stopping tablet replica
I20240502 14:06:13.990599 24174 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:14.021378 23343 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.022006 23189 consensus_queue.cc:237] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:14.031690 23926 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:14.033278 24199 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:14.043578 24130 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.046989 23265 raft_consensus.cc:1261] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:14.053236 24178 consensus_queue.cc:1035] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:14.054416 24213 raft_consensus.cc:2229] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.059607 24213 raft_consensus.cc:2258] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.059742 23269 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.068848 24213 ts_tablet_manager.cc:1875] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:14.072904 23091 catalog_manager.cc:5484] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 reported cstate change: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.068939 23265 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.071766 23927 raft_consensus.cc:2932] T 6c5069e564364a3b82e55d410fc03cec P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
W20240502 14:06:14.098765 23313 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.129.1:38171, user_credentials={real_user=slave}} blocked reactor thread for 61484us
I20240502 14:06:14.102677 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c5069e564364a3b82e55d410fc03cec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:14.112706 23269 consensus_queue.cc:237] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:14.117941 23189 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 67a108c8df8d451599ff168ac618d878 (127.21.129.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.120836 23092 catalog_manager.cc:5484] T 6c5069e564364a3b82e55d410fc03cec P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index 3 to 4, VOTER 980e09c2959e408db93d72b7125ccf63 (127.21.129.3) evicted. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:14.120934 24213 ts_tablet_manager.cc:1888] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:14.122502 24213 log.cc:1196] T 6132f4449de74e15a3cbe93f03a74b71 P 82490d85cd3d4a1180dc3ecabdd88598: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root/wals/6132f4449de74e15a3cbe93f03a74b71
I20240502 14:06:14.124361 23076 catalog_manager.cc:4830] TS 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): tablet 6132f4449de74e15a3cbe93f03a74b71 (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:14.128937 23927 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:14.130177 24064 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:14.138543 23339 raft_consensus.cc:1261] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:14.138484 23334 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5069e564364a3b82e55d410fc03cec with delete_type TABLET_DATA_TOMBSTONED (TS 980e09c2959e408db93d72b7125ccf63 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56144
I20240502 14:06:14.141372 24208 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.143535 24064 consensus_queue.cc:1035] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:14.144590 23927 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.152571 23076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b17aa98c3f04a58a5613d8b4af43f2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:14.153136 23339 raft_consensus.cc:2932] T 8b17aa98c3f04a58a5613d8b4af43f2e P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.164057 24215 tablet_replica.cc:330] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: stopping tablet replica
I20240502 14:06:14.174682 24215 raft_consensus.cc:2229] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.175410 24215 raft_consensus.cc:2258] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.177899 23090 catalog_manager.cc:5484] T 8b17aa98c3f04a58a5613d8b4af43f2e P 82490d85cd3d4a1180dc3ecabdd88598 reported cstate change: config changed from index 3 to 4, VOTER cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1) evicted. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:14.184108 23341 consensus_queue.cc:237] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } }
I20240502 14:06:14.186419 24215 ts_tablet_manager.cc:1875] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:14.190233 23928 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:14.191902 24202 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:14.192557 23197 raft_consensus.cc:1261] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:14.199156 24134 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.200641 24210 consensus_queue.cc:1035] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:14.206286 23197 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.213343 24215 ts_tablet_manager.cc:1888] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:14.215723 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b17aa98c3f04a58a5613d8b4af43f2e with delete_type TABLET_DATA_TOMBSTONED (TS cc4007c6e59b4361a155c5e8a79bbfb7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40056
I20240502 14:06:14.219497 23079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b400e3ddeee74798b411ee7a4e285b7a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:14.205595 23927 raft_consensus.cc:2932] T b400e3ddeee74798b411ee7a4e285b7a P 67a108c8df8d451599ff168ac618d878 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } } }
I20240502 14:06:14.225226 24215 log.cc:1196] T 6c5069e564364a3b82e55d410fc03cec P 980e09c2959e408db93d72b7125ccf63: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-2-root/wals/6c5069e564364a3b82e55d410fc03cec
I20240502 14:06:14.232869 23079 catalog_manager.cc:4830] TS 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179): tablet 6c5069e564364a3b82e55d410fc03cec (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:14.241705 24216 tablet_replica.cc:330] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: stopping tablet replica
I20240502 14:06:14.242914 24216 raft_consensus.cc:2229] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.243680 24216 raft_consensus.cc:2258] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.238508 23089 catalog_manager.cc:5484] T b400e3ddeee74798b411ee7a4e285b7a P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index 3 to 4, VOTER 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2) evicted. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:14.247143 24216 ts_tablet_manager.cc:1875] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:14.262769 23256 tablet_service.cc:1514] Processing DeleteTablet for tablet b400e3ddeee74798b411ee7a4e285b7a with delete_type TABLET_DATA_TOMBSTONED (TS 82490d85cd3d4a1180dc3ecabdd88598 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52056
I20240502 14:06:14.263756 24213 tablet_replica.cc:330] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: stopping tablet replica
I20240502 14:06:14.264874 24213 raft_consensus.cc:2229] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.265475 24213 raft_consensus.cc:2258] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.273898 24213 ts_tablet_manager.cc:1875] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:14.290196 24216 ts_tablet_manager.cc:1888] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:14.292536 24216 log.cc:1196] T 8b17aa98c3f04a58a5613d8b4af43f2e P cc4007c6e59b4361a155c5e8a79bbfb7: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-0-root/wals/8b17aa98c3f04a58a5613d8b4af43f2e
I20240502 14:06:14.294696 23079 catalog_manager.cc:4830] TS cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171): tablet 8b17aa98c3f04a58a5613d8b4af43f2e (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:14.301885 24213 ts_tablet_manager.cc:1888] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:14.302307 24213 log.cc:1196] T b400e3ddeee74798b411ee7a4e285b7a P 82490d85cd3d4a1180dc3ecabdd88598: Deleting WAL directory at /tmp/dist-test-taskk2SyS2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714658692869395-22020-0/minicluster-data/ts-1-root/wals/b400e3ddeee74798b411ee7a4e285b7a
I20240502 14:06:14.305217 23076 catalog_manager.cc:4830] TS 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169): tablet b400e3ddeee74798b411ee7a4e285b7a (table test-workload [id=b135b2923733409a98bf07d2863419d5]) successfully deleted
I20240502 14:06:15.292907 23269 consensus_queue.cc:237] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
W20240502 14:06:15.304111 23237 consensus_peers.cc:485] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbb27a4bdc15414cb260ffc2b673b5ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:15.304805 23189 raft_consensus.cc:1261] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.305358 23343 raft_consensus.cc:1261] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer 82490d85cd3d4a1180dc3ecabdd88598: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.306489 24219 consensus_queue.cc:1035] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.307309 24208 consensus_queue.cc:1035] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.314323 24219 raft_consensus.cc:2932] T fbb27a4bdc15414cb260ffc2b673b5ab P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.315618 23189 raft_consensus.cc:2932] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.316414 23339 raft_consensus.cc:2932] T fbb27a4bdc15414cb260ffc2b673b5ab P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.328528 23339 consensus_queue.cc:237] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:15.337057 23265 raft_consensus.cc:1261] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:06:15.337703 23312 consensus_peers.cc:485] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a9e5363c783444bbb4f631a0ee208d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:15.339398 23189 raft_consensus.cc:1261] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Refusing update from remote peer 980e09c2959e408db93d72b7125ccf63: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.340054 24220 consensus_queue.cc:1035] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.340940 24217 consensus_queue.cc:1035] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.340389 23089 catalog_manager.cc:5484] T fbb27a4bdc15414cb260ffc2b673b5ab P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "82490d85cd3d4a1180dc3ecabdd88598" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } attrs { replace: true } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.356329 24217 raft_consensus.cc:2932] T 9a9e5363c783444bbb4f631a0ee208d9 P 980e09c2959e408db93d72b7125ccf63 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.368301 23189 raft_consensus.cc:2932] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.369839 23265 raft_consensus.cc:2932] T 9a9e5363c783444bbb4f631a0ee208d9 P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.373171 23197 consensus_queue.cc:237] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:15.403393 23089 catalog_manager.cc:5484] T 9a9e5363c783444bbb4f631a0ee208d9 P cc4007c6e59b4361a155c5e8a79bbfb7 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "980e09c2959e408db93d72b7125ccf63" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } attrs { replace: true } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.407526 23269 raft_consensus.cc:1261] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.408515 23339 raft_consensus.cc:1261] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.410158 24218 consensus_queue.cc:1035] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.410975 24232 consensus_queue.cc:1035] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:06:15.415814 23161 consensus_peers.cc:485] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69a6ce78fbc0428199f226fd9120217f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:15.425885 24232 raft_consensus.cc:2932] T 69a6ce78fbc0428199f226fd9120217f P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.427796 23339 raft_consensus.cc:2932] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.430744 23265 raft_consensus.cc:2932] T 69a6ce78fbc0428199f226fd9120217f P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.441540 23189 consensus_queue.cc:237] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } }
I20240502 14:06:15.444794 23092 catalog_manager.cc:5484] T 69a6ce78fbc0428199f226fd9120217f P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
W20240502 14:06:15.472816 23161 consensus_peers.cc:485] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 -> Peer 67a108c8df8d451599ff168ac618d878 (127.21.129.4:46761): Couldn't send request to peer 67a108c8df8d451599ff168ac618d878. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50b2b38a668947049fc57d84a48e6bdb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:15.473373 23343 raft_consensus.cc:1261] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.474136 23265 raft_consensus.cc:1261] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Refusing update from remote peer cc4007c6e59b4361a155c5e8a79bbfb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.474583 24232 consensus_queue.cc:1035] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.475479 24218 consensus_queue.cc:1035] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:15.483462 24218 raft_consensus.cc:2932] T 50b2b38a668947049fc57d84a48e6bdb P cc4007c6e59b4361a155c5e8a79bbfb7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.493360 23343 raft_consensus.cc:2932] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.506160 23266 raft_consensus.cc:2932] T 50b2b38a668947049fc57d84a48e6bdb P 82490d85cd3d4a1180dc3ecabdd88598 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.508313 23089 catalog_manager.cc:5484] T 50b2b38a668947049fc57d84a48e6bdb P 980e09c2959e408db93d72b7125ccf63 reported cstate change: config changed from index -1 to 2, NON_VOTER 67a108c8df8d451599ff168ac618d878 (127.21.129.4) added. New cstate: current_term: 1 leader_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc4007c6e59b4361a155c5e8a79bbfb7" member_type: VOTER last_known_addr { host: "127.21.129.1" port: 38171 } } peers { permanent_uuid: "82490d85cd3d4a1180dc3ecabdd88598" member_type: VOTER last_known_addr { host: "127.21.129.2" port: 34169 } } peers { permanent_uuid: "980e09c2959e408db93d72b7125ccf63" member_type: VOTER last_known_addr { host: "127.21.129.3" port: 45179 } attrs { replace: true } } peers { permanent_uuid: "67a108c8df8d451599ff168ac618d878" member_type: NON_VOTER last_known_addr { host: "127.21.129.4" port: 46761 } attrs { promote: true } } }
I20240502 14:06:15.722551 24240 ts_tablet_manager.cc:902] T fbb27a4bdc15414cb260ffc2b673b5ab P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 82490d85cd3d4a1180dc3ecabdd88598 (127.21.129.2:34169)
F20240502 14:06:15.723635 24240 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-12b5382d476949269a01d25f2e60c2d8.queue_time_us, tablet-download-pool-12b5382d476949269a01d25f2e60c2d8.queue_time_us) (tablet-download-pool-12b5382d476949269a01d25f2e60c2d8.run_cpu_time_us, tablet-download-pool-12b5382d476949269a01d25f2e60c2d8.run_cpu_time_us) (tablet-download-pool-12b5382d476949269a01d25f2e60c2d8.run_wall_time_us, tablet-download-pool-12b5382d476949269a01d25f2e60c2d8.run_wall_time_us) (tablet-download-pool-5f6c31e65498481696374e87a81890fa.queue_time_us, tablet-download-pool-5f6c31e65498481696374e87a81890fa.queue_time_us) (tablet-download-pool-5f6c31e65498481696374e87a81890fa.run_cpu_time_us, tablet-download-pool-5f6c31e65498481696374e87a81890fa.run_cpu_time_us) (tablet-download-pool-5f6c31e65498481696374e87a81890fa.run_wall_time_us, tablet-download-pool-5f6c31e65498481696374e87a81890fa.run_wall_time_us) (tablet-download-pool-6132f4449de74e15a3cbe93f03a74b71.queue_time_us, tablet-download-pool-6132f4449de74e15a3cbe93f03a74b71.queue_time_us) (tablet-download-pool-6132f4449de74e15a3cbe93f03a74b71.run_cpu_time_us, tablet-download-pool-6132f4449de74e15a3cbe93f03a74b71.run_cpu_time_us) (tablet-download-pool-6132f4449de74e15a3cbe93f03a74b71.run_wall_time_us, tablet-download-pool-6132f4449de74e15a3cbe93f03a74b71.run_wall_time_us) (tablet-download-pool-6b0ebc495868467d95f7e6b636732483.queue_time_us, tablet-download-pool-6b0ebc495868467d95f7e6b636732483.queue_time_us) (tablet-download-pool-6b0ebc495868467d95f7e6b636732483.run_cpu_time_us, tablet-download-pool-6b0ebc495868467d95f7e6b636732483.run_cpu_time_us) (tablet-download-pool-6b0ebc495868467d95f7e6b636732483.run_wall_time_us, tablet-download-pool-6b0ebc495868467d95f7e6b636732483.run_wall_time_us) (tablet-download-pool-6c123d564235427795f0271d9ae4213c.queue_time_us, tablet-download-pool-6c123d564235427795f0271d9ae4213c.queue_time_us) (tablet-download-pool-6c123d564235427795f0271d9ae4213c.run_cpu_time_us, tablet-download-pool-6c123d564235427795f0271d9ae4213c.run_cpu_time_us) (tablet-download-pool-6c123d564235427795f0271d9ae4213c.run_wall_time_us, tablet-download-pool-6c123d564235427795f0271d9ae4213c.run_wall_time_us) (tablet-download-pool-6c5069e564364a3b82e55d410fc03cec.queue_time_us, tablet-download-pool-6c5069e564364a3b82e55d410fc03cec.queue_time_us) (tablet-download-pool-6c5069e564364a3b82e55d410fc03cec.run_cpu_time_us, tablet-download-pool-6c5069e564364a3b82e55d410fc03cec.run_cpu_time_us) (tablet-download-pool-6c5069e564364a3b82e55d410fc03cec.run_wall_time_us, tablet-download-pool-6c5069e564364a3b82e55d410fc03cec.run_wall_time_us) (tablet-download-pool-8b17aa98c3f04a58a5613d8b4af43f2e.queue_time_us, tablet-download-pool-8b17aa98c3f04a58a5613d8b4af43f2e.queue_time_us) (tablet-download-pool-8b17aa98c3f04a58a5613d8b4af43f2e.run_cpu_time_us, tablet-download-pool-8b17aa98c3f04a58a5613d8b4af43f2e.run_cpu_time_us) (tablet-download-pool-8b17aa98c3f04a58a5613d8b4af43f2e.run_wall_time_us, tablet-download-pool-8b17aa98c3f04a58a5613d8b4af43f2e.run_wall_time_us) (tablet-download-pool-afca5b05bac0423ca02254a903b07eca.queue_time_us, tablet-download-pool-afca5b05bac0423ca02254a903b07eca.queue_time_us) (tablet-download-pool-afca5b05bac0423ca02254a903b07eca.run_cpu_time_us, tablet-download-pool-afca5b05bac0423ca02254a903b07eca.run_cpu_time_us) (tablet-download-pool-afca5b05bac0423ca02254a903b07eca.run_wall_time_us, tablet-download-pool-afca5b05bac0423ca02254a903b07eca.run_wall_time_us) (tablet-download-pool-b400e3ddeee74798b411ee7a4e285b7a.queue_time_us, tablet-download-pool-b400e3ddeee74798b411ee7a4e285b7a.queue_time_us) (tablet-download-pool-b400e3ddeee74798b411ee7a4e285b7a.run_cpu_time_us, tablet-download-pool-b400e3ddeee74798b411ee7a4e285b7a.run_cpu_time_us) (tablet-download-pool-b400e3ddeee74798b411ee7a4e285b7a.run_wall_time_us, tablet-download-pool-b400e3ddeee74798b411ee7a4e285b7a.run_wall_time_us) (tablet-download-pool-b90f20f57cf34057a5e3301a044ad3ca.queue_time_us, tablet-download-pool-b90f20f57cf34057a5e3301a044ad3ca.queue_time_us) (tablet-download-pool-b90f20f57cf34057a5e3301a044ad3ca.run_cpu_time_us, tablet-download-pool-b90f20f57cf34057a5e3301a044ad3ca.run_cpu_time_us) (tablet-download-pool-b90f20f57cf34057a5e3301a044ad3ca.run_wall_time_us, tablet-download-pool-b90f20f57cf34057a5e3301a044ad3ca.run_wall_time_us) (tablet-download-pool-cccf9e842a284cf8bcc8d60d0dd1a22b.queue_time_us, tablet-download-pool-cccf9e842a284cf8bcc8d60d0dd1a22b.queue_time_us) (tablet-download-pool-cccf9e842a284cf8bcc8d60d0dd1a22b.run_cpu_time_us, tablet-download-pool-cccf9e842a284cf8bcc8d60d0dd1a22b.run_cpu_time_us) (tablet-download-pool-cccf9e842a284cf8bcc8d60d0dd1a22b.run_wall_time_us, tablet-download-pool-cccf9e842a284cf8bcc8d60d0dd1a22b.run_wall_time_us) (tablet-download-pool-eb3ff7f6ffd34055abb266d2b25e2e4d.queue_time_us, tablet-download-pool-eb3ff7f6ffd34055abb266d2b25e2e4d.queue_time_us) (tablet-download-pool-eb3ff7f6ffd34055abb266d2b25e2e4d.run_cpu_time_us, tablet-download-pool-eb3ff7f6ffd34055abb266d2b25e2e4d.run_cpu_time_us) (tablet-download-pool-eb3ff7f6ffd34055abb266d2b25e2e4d.run_wall_time_us, tablet-download-pool-eb3ff7f6ffd34055abb266d2b25e2e4d.run_wall_time_us) (tablet-download-pool-fbb27a4bdc15414cb260ffc2b673b5ab.queue_time_us, tablet-download-pool-fbb27a4bdc15414cb260ffc2b673b5ab.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714658775 (unix time) try "date -d @1714658775" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005604) received by PID 22020 (TID 0x7f670056e700) from PID 22020; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f67cebe0980 (unknown) at ??:0
@ 0x7f67c8154fb7 gsignal at ??:0
@ 0x7f67c8156921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f67caf4ada4 google::LogMessage::SendToLog() at ??:0
@ 0x7f67caf4b950 google::LogMessage::Flush() at ??:0
@ 0x7f67caf50a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:06:15.888958 24247 ts_tablet_manager.cc:902] T 69a6ce78fbc0428199f226fd9120217f P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
@ 0x7f67cb8d83c7 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:06:15.898329 24248 ts_tablet_manager.cc:902] T 50b2b38a668947049fc57d84a48e6bdb P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer cc4007c6e59b4361a155c5e8a79bbfb7 (127.21.129.1:38171)
@ 0x7f67cb8ae94f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f67cb8ac0a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:06:15.936133 24253 ts_tablet_manager.cc:902] T 9a9e5363c783444bbb4f631a0ee208d9 P 67a108c8df8d451599ff168ac618d878: Initiating tablet copy from peer 980e09c2959e408db93d72b7125ccf63 (127.21.129.3:45179)
@ 0x7f67d36cc422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f67d36d2c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f67d37684ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f67d37993cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f67d377f1ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f67d377f17a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7f67d377f10a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f67d377f0d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f67d377e2fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f67d4b24da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f67d4b24bda std::__1::function<>::operator()() at ??:0
@ 0x7f67cb8b0237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f67cb8b3692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f67cb8b364a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f67cb8b35da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f67cb8b35a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f67cb8b289e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f67d4b24da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f67d4b24bda std::__1::function<>::operator()() at ??:0
@ 0x7f67cb892937 kudu::Thread::SuperviseThread() at ??:0