[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:48:45.445925 17046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.165.190:42449
I20240502 01:48:45.447327 17046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:45.448511 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:45.465502 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:45.465942 17053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:45.466763 17046 server_base.cc:1034] running on GCE node
W20240502 01:48:45.466506 17055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:46.543264 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:46.543555 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:46.543756 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614526543735 us; error 0 us; skew 500 ppm
I20240502 01:48:46.544638 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:46.550014 17046 webserver.cc:460] Webserver started at http://127.16.165.190:43305/ using document root <none> and password file <none>
I20240502 01:48:46.550886 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:46.551110 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:46.551530 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:46.558357 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/master-0-root/instance:
uuid: "211216f22b6a415f83466576fe691493"
format_stamp: "Formatted at 2024-05-02 01:48:46 on dist-test-slave-kj4b"
I20240502 01:48:46.568255 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20240502 01:48:46.573628 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:46.574765 17046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:48:46.575114 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/master-0-root
uuid: "211216f22b6a415f83466576fe691493"
format_stamp: "Formatted at 2024-05-02 01:48:46 on dist-test-slave-kj4b"
I20240502 01:48:46.575496 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:46.635488 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:46.637094 17046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:46.637578 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:46.709429 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.190:42449
I20240502 01:48:46.709527 17113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.190:42449 every 8 connection(s)
I20240502 01:48:46.716531 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:46.740372 17114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493: Bootstrap starting.
I20240502 01:48:46.745962 17114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:46.747804 17114 log.cc:824] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:46.752416 17114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493: No bootstrap required, opened a new log
I20240502 01:48:46.777415 17114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211216f22b6a415f83466576fe691493" member_type: VOTER }
I20240502 01:48:46.777987 17114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:46.778198 17114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211216f22b6a415f83466576fe691493, State: Initialized, Role: FOLLOWER
I20240502 01:48:46.778952 17114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211216f22b6a415f83466576fe691493" member_type: VOTER }
I20240502 01:48:46.779428 17114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:46.779659 17114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:46.779984 17114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:46.851936 17114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211216f22b6a415f83466576fe691493" member_type: VOTER }
I20240502 01:48:46.852809 17114 leader_election.cc:302] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211216f22b6a415f83466576fe691493; no voters:
I20240502 01:48:46.854732 17114 leader_election.cc:288] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:46.855093 17117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:46.857631 17117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 1 LEADER]: Becoming Leader. State: Replica: 211216f22b6a415f83466576fe691493, State: Running, Role: LEADER
I20240502 01:48:46.858580 17117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211216f22b6a415f83466576fe691493" member_type: VOTER }
I20240502 01:48:46.859587 17114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:46.869462 17118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211216f22b6a415f83466576fe691493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211216f22b6a415f83466576fe691493" member_type: VOTER } }
I20240502 01:48:46.870594 17118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:46.871688 17119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211216f22b6a415f83466576fe691493. Latest consensus state: current_term: 1 leader_uuid: "211216f22b6a415f83466576fe691493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211216f22b6a415f83466576fe691493" member_type: VOTER } }
I20240502 01:48:46.872321 17119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:46.879464 17127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:48:46.891728 17127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:48:46.894174 17046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:48:46.907696 17127 catalog_manager.cc:1347] Generated new cluster ID: 52120aed9d4447c8a60f6221aef89eea
I20240502 01:48:46.908044 17127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:46.935600 17127 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:48:46.937844 17127 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:46.958102 17127 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493: Generated new TSK 0
I20240502 01:48:46.959352 17127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:48:47.025373 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:47.033668 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:47.034906 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:47.039069 17046 server_base.cc:1034] running on GCE node
W20240502 01:48:47.039553 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:47.040468 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:47.040674 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:47.040849 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614527040828 us; error 0 us; skew 500 ppm
I20240502 01:48:47.041437 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:47.045861 17046 webserver.cc:460] Webserver started at http://127.16.165.129:34201/ using document root <none> and password file <none>
I20240502 01:48:47.046416 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:47.046615 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:47.046895 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:47.048122 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-0-root/instance:
uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
format_stamp: "Formatted at 2024-05-02 01:48:47 on dist-test-slave-kj4b"
I20240502 01:48:47.053680 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 01:48:47.057987 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.058784 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:48:47.059072 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-0-root
uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
format_stamp: "Formatted at 2024-05-02 01:48:47 on dist-test-slave-kj4b"
I20240502 01:48:47.059365 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:47.078233 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:47.079558 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:47.081990 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:47.086697 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:47.087041 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.087353 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:47.087530 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.185006 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.129:44677
I20240502 01:48:47.185076 17208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.129:44677 every 8 connection(s)
I20240502 01:48:47.192108 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:47.209298 17213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:47.210503 17214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:47.218540 17216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:47.219233 17046 server_base.cc:1034] running on GCE node
I20240502 01:48:47.220217 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:47.220499 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:47.220724 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614527220706 us; error 0 us; skew 500 ppm
I20240502 01:48:47.221554 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:47.222259 17209 heartbeater.cc:347] Connected to a master server at 127.16.165.190:42449
I20240502 01:48:47.222667 17209 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:47.223685 17209 heartbeater.cc:511] Master 127.16.165.190:42449 requested a full tablet report, sending...
I20240502 01:48:47.224784 17046 webserver.cc:460] Webserver started at http://127.16.165.130:39743/ using document root <none> and password file <none>
I20240502 01:48:47.225463 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:47.225731 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:47.226065 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:47.226631 17079 ts_manager.cc:194] Registered new tserver with Master: 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677)
I20240502 01:48:47.227551 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-1-root/instance:
uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
format_stamp: "Formatted at 2024-05-02 01:48:47 on dist-test-slave-kj4b"
I20240502 01:48:47.228865 17079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37892
I20240502 01:48:47.234448 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 01:48:47.239415 17221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.240257 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:48:47.240528 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-1-root
uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
format_stamp: "Formatted at 2024-05-02 01:48:47 on dist-test-slave-kj4b"
I20240502 01:48:47.240788 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:47.268805 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:47.270028 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:47.271306 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:47.274616 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:47.274816 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.275033 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:47.275182 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.369004 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.130:44481
I20240502 01:48:47.369141 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.130:44481 every 8 connection(s)
I20240502 01:48:47.376085 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:47.383527 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:47.388881 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:47.390609 17284 heartbeater.cc:347] Connected to a master server at 127.16.165.190:42449
I20240502 01:48:47.391080 17284 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:47.392066 17284 heartbeater.cc:511] Master 127.16.165.190:42449 requested a full tablet report, sending...
I20240502 01:48:47.392984 17046 server_base.cc:1034] running on GCE node
I20240502 01:48:47.394189 17079 ts_manager.cc:194] Registered new tserver with Master: b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
W20240502 01:48:47.394243 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:47.395390 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:47.395639 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:47.395735 17079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37898
I20240502 01:48:47.395834 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614527395819 us; error 0 us; skew 500 ppm
I20240502 01:48:47.397135 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:47.401582 17046 webserver.cc:460] Webserver started at http://127.16.165.131:35869/ using document root <none> and password file <none>
I20240502 01:48:47.402045 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:47.402233 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:47.402472 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:47.403503 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-2-root/instance:
uuid: "3c83f4847492417a91fb69cc7afc616a"
format_stamp: "Formatted at 2024-05-02 01:48:47 on dist-test-slave-kj4b"
I20240502 01:48:47.408288 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240502 01:48:47.411902 17295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.412734 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:48:47.413013 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-2-root
uuid: "3c83f4847492417a91fb69cc7afc616a"
format_stamp: "Formatted at 2024-05-02 01:48:47 on dist-test-slave-kj4b"
I20240502 01:48:47.413290 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714614525285352-17046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:47.444734 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:47.445969 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:47.448218 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:47.451431 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:47.451634 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:47.451887 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:47.452055 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 01:48:47.541033 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.131:36061
I20240502 01:48:47.541119 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.131:36061 every 8 connection(s)
I20240502 01:48:47.563072 17359 heartbeater.cc:347] Connected to a master server at 127.16.165.190:42449
I20240502 01:48:47.563439 17359 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:47.564255 17359 heartbeater.cc:511] Master 127.16.165.190:42449 requested a full tablet report, sending...
I20240502 01:48:47.566366 17079 ts_manager.cc:194] Registered new tserver with Master: 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:47.566515 17046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014864668s
I20240502 01:48:47.568257 17079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37906
I20240502 01:48:47.569973 17046 test_util.cc:268] Using random seed: -841502698
I20240502 01:48:47.621068 17079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37910:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:48:47.624843 17079 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:48:47.716364 17323 tablet_service.cc:1467] Processing CreateTablet for tablet 8e8576363f2f44f7b349932695b7fa58 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 01:48:47.718073 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e8576363f2f44f7b349932695b7fa58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.725443 17320 tablet_service.cc:1467] Processing CreateTablet for tablet d761bc3026844a54822528ce6e783df8 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 01:48:47.727969 17322 tablet_service.cc:1467] Processing CreateTablet for tablet 98e78e9fa9e74827839975255f3f4d3b (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 01:48:47.727221 17321 tablet_service.cc:1467] Processing CreateTablet for tablet 5260c565a7ea4e06ab2c9ea242e287f4 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 01:48:47.731652 17319 tablet_service.cc:1467] Processing CreateTablet for tablet b067abdcee08465284065c9f6a221414 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 01:48:47.732381 17318 tablet_service.cc:1467] Processing CreateTablet for tablet e7738ff80ca94c59bfe1bcd5f3c0bc73 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 01:48:47.733547 17320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761bc3026844a54822528ce6e783df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.737156 17322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e78e9fa9e74827839975255f3f4d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.744506 17317 tablet_service.cc:1467] Processing CreateTablet for tablet b0ca0cf4837b44489774215a096ac335 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 01:48:47.746466 17318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7738ff80ca94c59bfe1bcd5f3c0bc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.748127 17317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ca0cf4837b44489774215a096ac335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.750332 17321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5260c565a7ea4e06ab2c9ea242e287f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.752944 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b067abdcee08465284065c9f6a221414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.770591 17169 tablet_service.cc:1467] Processing CreateTablet for tablet b067abdcee08465284065c9f6a221414 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 01:48:47.782202 17169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b067abdcee08465284065c9f6a221414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.767496 17174 tablet_service.cc:1467] Processing CreateTablet for tablet 8e8576363f2f44f7b349932695b7fa58 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 01:48:47.768821 17173 tablet_service.cc:1467] Processing CreateTablet for tablet 98e78e9fa9e74827839975255f3f4d3b (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 01:48:47.788729 17172 tablet_service.cc:1467] Processing CreateTablet for tablet 5260c565a7ea4e06ab2c9ea242e287f4 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 01:48:47.790676 17168 tablet_service.cc:1467] Processing CreateTablet for tablet e7738ff80ca94c59bfe1bcd5f3c0bc73 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 01:48:47.767176 17171 tablet_service.cc:1467] Processing CreateTablet for tablet d761bc3026844a54822528ce6e783df8 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 01:48:47.794785 17316 tablet_service.cc:1467] Processing CreateTablet for tablet 55676801f633492291856a8cd6c61819 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 01:48:47.796519 17316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55676801f633492291856a8cd6c61819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.798522 17167 tablet_service.cc:1467] Processing CreateTablet for tablet b0ca0cf4837b44489774215a096ac335 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 01:48:47.784148 17166 tablet_service.cc:1467] Processing CreateTablet for tablet 55676801f633492291856a8cd6c61819 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 01:48:47.806449 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5260c565a7ea4e06ab2c9ea242e287f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.809687 17174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e8576363f2f44f7b349932695b7fa58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.819594 17165 tablet_service.cc:1467] Processing CreateTablet for tablet 097fbd59fa8b49ad85f0b004b3cbcd9d (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 01:48:47.823204 17166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55676801f633492291856a8cd6c61819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.825913 17165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097fbd59fa8b49ad85f0b004b3cbcd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.834408 17171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761bc3026844a54822528ce6e783df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.836875 17167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ca0cf4837b44489774215a096ac335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.841912 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7738ff80ca94c59bfe1bcd5f3c0bc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.847445 17173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e78e9fa9e74827839975255f3f4d3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:48:47.876207 17126 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:48:47.900146 17246 tablet_service.cc:1467] Processing CreateTablet for tablet d761bc3026844a54822528ce6e783df8 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 01:48:47.901118 17249 tablet_service.cc:1467] Processing CreateTablet for tablet 8e8576363f2f44f7b349932695b7fa58 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 01:48:47.901496 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761bc3026844a54822528ce6e783df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.923939 17249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e8576363f2f44f7b349932695b7fa58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.930213 17244 tablet_service.cc:1467] Processing CreateTablet for tablet e7738ff80ca94c59bfe1bcd5f3c0bc73 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 01:48:47.934891 17243 tablet_service.cc:1467] Processing CreateTablet for tablet b0ca0cf4837b44489774215a096ac335 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 01:48:47.935065 17242 tablet_service.cc:1467] Processing CreateTablet for tablet 55676801f633492291856a8cd6c61819 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 01:48:47.935941 17245 tablet_service.cc:1467] Processing CreateTablet for tablet b067abdcee08465284065c9f6a221414 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 01:48:47.938477 17248 tablet_service.cc:1467] Processing CreateTablet for tablet 98e78e9fa9e74827839975255f3f4d3b (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 01:48:47.939182 17247 tablet_service.cc:1467] Processing CreateTablet for tablet 5260c565a7ea4e06ab2c9ea242e287f4 (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 01:48:47.947333 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7738ff80ca94c59bfe1bcd5f3c0bc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.932484 17241 tablet_service.cc:1467] Processing CreateTablet for tablet 097fbd59fa8b49ad85f0b004b3cbcd9d (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 01:48:47.970834 17247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5260c565a7ea4e06ab2c9ea242e287f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.972432 17377 tablet_bootstrap.cc:492] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:47.980708 17248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e78e9fa9e74827839975255f3f4d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.985229 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b067abdcee08465284065c9f6a221414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.986447 17243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ca0cf4837b44489774215a096ac335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.981786 17323 tablet_service.cc:1467] Processing CreateTablet for tablet 097fbd59fa8b49ad85f0b004b3cbcd9d (DEFAULT_TABLE table=test-workload [id=abeafc6de5fd4833bfdb790ee5ecca18]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 01:48:47.988497 17241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097fbd59fa8b49ad85f0b004b3cbcd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.988481 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097fbd59fa8b49ad85f0b004b3cbcd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:47.992322 17242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55676801f633492291856a8cd6c61819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:48.057144 17378 tablet_bootstrap.cc:492] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.095979 17377 tablet_bootstrap.cc:654] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.131610 17377 tablet_bootstrap.cc:492] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.133008 17377 ts_tablet_manager.cc:1367] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.204s user 0.012s sys 0.008s
I20240502 01:48:48.137310 17377 raft_consensus.cc:348] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.142722 17377 raft_consensus.cc:374] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.145186 17377 raft_consensus.cc:729] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.145435 17378 tablet_bootstrap.cc:654] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.146322 17377 consensus_queue.cc:260] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.151082 17379 tablet_bootstrap.cc:492] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.155611 17359 heartbeater.cc:503] Master 127.16.165.190:42449 was elected leader, sending a full tablet report...
I20240502 01:48:48.162138 17377 ts_tablet_manager.cc:1398] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.028s user 0.010s sys 0.000s
I20240502 01:48:48.163528 17377 tablet_bootstrap.cc:492] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.180238 17377 tablet_bootstrap.cc:654] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.179690 17378 tablet_bootstrap.cc:492] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.181799 17378 ts_tablet_manager.cc:1367] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.125s user 0.015s sys 0.005s
I20240502 01:48:48.180893 17379 tablet_bootstrap.cc:654] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.192191 17378 raft_consensus.cc:348] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.193029 17378 raft_consensus.cc:374] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.193338 17378 raft_consensus.cc:729] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.194056 17378 consensus_queue.cc:260] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.194846 17377 tablet_bootstrap.cc:492] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.195377 17377 ts_tablet_manager.cc:1367] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.000s
I20240502 01:48:48.198293 17377 raft_consensus.cc:348] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.199077 17377 raft_consensus.cc:374] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.199431 17377 raft_consensus.cc:729] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.200271 17209 heartbeater.cc:503] Master 127.16.165.190:42449 was elected leader, sending a full tablet report...
I20240502 01:48:48.200256 17377 consensus_queue.cc:260] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.201931 17377 ts_tablet_manager.cc:1398] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:48:48.202636 17377 tablet_bootstrap.cc:492] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.206849 17378 ts_tablet_manager.cc:1398] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20240502 01:48:48.208258 17377 tablet_bootstrap.cc:654] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.209314 17379 tablet_bootstrap.cc:492] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.209973 17379 ts_tablet_manager.cc:1367] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.014s
I20240502 01:48:48.213064 17379 raft_consensus.cc:348] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.213755 17379 raft_consensus.cc:374] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.214468 17385 raft_consensus.cc:482] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.214931 17385 raft_consensus.cc:504] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.215514 17377 tablet_bootstrap.cc:492] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.215988 17377 ts_tablet_manager.cc:1367] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240502 01:48:48.218583 17377 raft_consensus.cc:348] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.219326 17377 raft_consensus.cc:374] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.219643 17377 raft_consensus.cc:729] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.220333 17377 consensus_queue.cc:260] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.227208 17377 ts_tablet_manager.cc:1398] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:48:48.227788 17378 tablet_bootstrap.cc:492] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.228262 17377 tablet_bootstrap.cc:492] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.214071 17379 raft_consensus.cc:729] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.233726 17379 consensus_queue.cc:260] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.218806 17385 leader_election.cc:288] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.238909 17377 tablet_bootstrap.cc:654] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.239285 17379 ts_tablet_manager.cc:1398] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.029s user 0.004s sys 0.002s
I20240502 01:48:48.241587 17381 raft_consensus.cc:482] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.242184 17381 raft_consensus.cc:504] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.245010 17381 leader_election.cc:288] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.246950 17284 heartbeater.cc:503] Master 127.16.165.190:42449 was elected leader, sending a full tablet report...
I20240502 01:48:48.251533 17379 tablet_bootstrap.cc:492] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.264994 17381 raft_consensus.cc:482] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.266321 17381 raft_consensus.cc:504] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.268824 17381 leader_election.cc:288] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677)
I20240502 01:48:48.269042 17379 tablet_bootstrap.cc:654] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.271103 17378 tablet_bootstrap.cc:654] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.286417 17377 tablet_bootstrap.cc:492] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.287094 17377 ts_tablet_manager.cc:1367] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.059s user 0.021s sys 0.030s
I20240502 01:48:48.286893 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a" is_pre_election: true
W20240502 01:48:48.290534 17147 leader_election.cc:341] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:48:48.303516 17360 tablet.cc:2369] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:48:48.312657 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
W20240502 01:48:48.316535 17148 leader_election.cc:341] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.318213 17377 raft_consensus.cc:348] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.319521 17377 raft_consensus.cc:374] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.320976 17377 raft_consensus.cc:729] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.321698 17377 consensus_queue.cc:260] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.320279 17148 leader_election.cc:302] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:48.323804 17385 raft_consensus.cc:2726] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:48.324764 17377 ts_tablet_manager.cc:1398] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.037s user 0.006s sys 0.015s
I20240502 01:48:48.325745 17377 tablet_bootstrap.cc:492] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.333375 17377 tablet_bootstrap.cc:654] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.341409 17379 tablet_bootstrap.cc:492] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.341415 17378 tablet_bootstrap.cc:492] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.342036 17379 ts_tablet_manager.cc:1367] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.019s
I20240502 01:48:48.342123 17378 ts_tablet_manager.cc:1367] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.012s
I20240502 01:48:48.346108 17378 raft_consensus.cc:348] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.346876 17378 raft_consensus.cc:374] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.347159 17378 raft_consensus.cc:729] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.348369 17379 raft_consensus.cc:348] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.348645 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.348896 17258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.349285 17379 raft_consensus.cc:374] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.347853 17378 consensus_queue.cc:260] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.349920 17379 raft_consensus.cc:729] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.349990 17258 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.350831 17379 consensus_queue.cc:260] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.351893 17378 ts_tablet_manager.cc:1398] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240502 01:48:48.353082 17259 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.354058 17299 leader_election.cc:302] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:48.352825 17378 tablet_bootstrap.cc:492] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.354954 17381 raft_consensus.cc:2781] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.355407 17381 raft_consensus.cc:482] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.355780 17381 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.353520 17379 ts_tablet_manager.cc:1398] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:48:48.359460 17379 tablet_bootstrap.cc:492] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.361871 17299 leader_election.cc:302] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:48.372484 17377 tablet_bootstrap.cc:492] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.374732 17377 ts_tablet_manager.cc:1367] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.011s
I20240502 01:48:48.378651 17377 raft_consensus.cc:348] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.380391 17377 raft_consensus.cc:374] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.380978 17378 tablet_bootstrap.cc:654] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.370296 17381 raft_consensus.cc:504] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.370241 17399 raft_consensus.cc:2781] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.381987 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.382357 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.380755 17377 raft_consensus.cc:729] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.383648 17399 raft_consensus.cc:482] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.384114 17399 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:48:48.384797 17297 leader_election.cc:341] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.384027 17377 consensus_queue.cc:260] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
W20240502 01:48:48.385805 17297 leader_election.cc:341] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.388955 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.389619 17259 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.391219 17381 leader_election.cc:288] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677)
I20240502 01:48:48.393985 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
W20240502 01:48:48.395730 17297 leader_election.cc:341] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.396986 17259 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.397101 17381 raft_consensus.cc:482] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.400370 17381 raft_consensus.cc:504] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.400951 17299 leader_election.cc:302] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters: 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:48.402374 17381 leader_election.cc:288] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.396998 17379 tablet_bootstrap.cc:654] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.398650 17399 raft_consensus.cc:504] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.403323 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.403148 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.403963 17400 raft_consensus.cc:2781] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.404129 17183 raft_consensus.cc:2454] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.406208 17297 leader_election.cc:302] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:48:48.406457 17399 leader_election.cc:288] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.410377 17377 ts_tablet_manager.cc:1398] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20240502 01:48:48.411448 17378 tablet_bootstrap.cc:492] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
W20240502 01:48:48.411355 17299 leader_election.cc:341] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.412098 17378 ts_tablet_manager.cc:1367] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.009s
I20240502 01:48:48.412421 17399 raft_consensus.cc:2781] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.412796 17399 raft_consensus.cc:482] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.413151 17399 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.412205 17377 tablet_bootstrap.cc:492] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.404616 17400 raft_consensus.cc:686] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:48.415098 17378 raft_consensus.cc:348] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.416000 17378 raft_consensus.cc:374] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.416390 17378 raft_consensus.cc:729] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.417685 17378 consensus_queue.cc:260] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.415467 17400 consensus_queue.cc:237] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.420058 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.420656 17259 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.428233 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:48.429109 17377 tablet_bootstrap.cc:654] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
W20240502 01:48:48.432400 17297 leader_election.cc:341] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.426779 17259 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.432065 17399 raft_consensus.cc:504] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.434262 17379 tablet_bootstrap.cc:492] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.435566 17378 ts_tablet_manager.cc:1398] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20240502 01:48:48.435047 17379 ts_tablet_manager.cc:1367] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.003s
I20240502 01:48:48.436355 17299 leader_election.cc:302] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters: 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:48.437450 17378 tablet_bootstrap.cc:492] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.436070 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:48.436507 17399 leader_election.cc:288] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.440191 17402 raft_consensus.cc:2781] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.441183 17402 raft_consensus.cc:686] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:48.442395 17402 consensus_queue.cc:237] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.442231 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.445876 17379 raft_consensus.cc:348] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.446674 17379 raft_consensus.cc:374] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.447026 17379 raft_consensus.cc:729] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.441615 17183 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:48:48.452646 17299 leader_election.cc:341] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.448766 17379 consensus_queue.cc:260] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.455178 17390 raft_consensus.cc:482] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.456087 17390 raft_consensus.cc:504] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.460634 17390 leader_election.cc:288] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.467666 17183 raft_consensus.cc:2454] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.475298 17379 ts_tablet_manager.cc:1398] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.038s user 0.006s sys 0.002s
I20240502 01:48:48.469753 17297 leader_election.cc:302] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters: b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:48.477596 17379 tablet_bootstrap.cc:492] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.479482 17402 raft_consensus.cc:2781] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.480685 17402 raft_consensus.cc:686] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:48.480576 17079 catalog_manager.cc:5484] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:48.481986 17402 consensus_queue.cc:237] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.484841 17377 tablet_bootstrap.cc:492] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.485316 17377 ts_tablet_manager.cc:1367] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.073s user 0.025s sys 0.017s
I20240502 01:48:48.489045 17379 tablet_bootstrap.cc:654] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.496354 17378 tablet_bootstrap.cc:654] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.496802 17377 raft_consensus.cc:348] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.500890 17377 raft_consensus.cc:374] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.501487 17377 raft_consensus.cc:729] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.502383 17377 consensus_queue.cc:260] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.510823 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a" is_pre_election: true
I20240502 01:48:48.511523 17378 tablet_bootstrap.cc:492] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.514391 17333 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 0.
I20240502 01:48:48.517628 17224 leader_election.cc:302] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:48.517679 17377 ts_tablet_manager.cc:1398] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.031s user 0.007s sys 0.002s
I20240502 01:48:48.518774 17390 raft_consensus.cc:2781] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.518836 17378 ts_tablet_manager.cc:1367] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.082s user 0.015s sys 0.004s
I20240502 01:48:48.520821 17390 raft_consensus.cc:482] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.521270 17390 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.523154 17377 tablet_bootstrap.cc:492] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.530372 17390 raft_consensus.cc:504] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.533810 17378 raft_consensus.cc:348] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.534556 17378 raft_consensus.cc:374] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.534859 17378 raft_consensus.cc:729] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.537128 17378 consensus_queue.cc:260] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.539438 17378 ts_tablet_manager.cc:1398] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240502 01:48:48.541611 17390 leader_election.cc:288] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.542065 17377 tablet_bootstrap.cc:654] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.542726 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:48.543347 17333 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.543474 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.544828 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:48.548300 17378 tablet_bootstrap.cc:492] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
W20240502 01:48:48.550115 17223 leader_election.cc:341] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:48:48.551237 17223 leader_election.cc:341] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.559288 17333 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 1.
I20240502 01:48:48.560730 17224 leader_election.cc:302] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters: 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:48.561976 17390 raft_consensus.cc:2781] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.563676 17385 raft_consensus.cc:482] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.564217 17385 raft_consensus.cc:504] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.566247 17385 leader_election.cc:288] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.567023 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.568018 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a" is_pre_election: true
W20240502 01:48:48.568495 17148 leader_election.cc:341] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.568652 17333 raft_consensus.cc:2454] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 0.
I20240502 01:48:48.569829 17147 leader_election.cc:302] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters: b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:48.574352 17385 raft_consensus.cc:2781] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.574892 17385 raft_consensus.cc:482] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.575232 17385 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.578692 17078 catalog_manager.cc:5484] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:48.581009 17078 catalog_manager.cc:5484] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:48.581827 17385 raft_consensus.cc:504] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.582310 17377 tablet_bootstrap.cc:492] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.582926 17377 ts_tablet_manager.cc:1367] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.000s
I20240502 01:48:48.584616 17385 leader_election.cc:288] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 election: Requested vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.584540 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.585651 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
W20240502 01:48:48.585856 17148 leader_election.cc:341] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:48.586241 17333 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.570554 17379 tablet_bootstrap.cc:492] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.589125 17379 ts_tablet_manager.cc:1367] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.112s user 0.019s sys 0.012s
I20240502 01:48:48.587949 17377 raft_consensus.cc:348] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.589576 17377 raft_consensus.cc:374] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.589901 17377 raft_consensus.cc:729] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.590759 17377 consensus_queue.cc:260] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.589793 17378 tablet_bootstrap.cc:654] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.593518 17377 ts_tablet_manager.cc:1398] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 01:48:48.594475 17377 tablet_bootstrap.cc:492] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.591763 17379 raft_consensus.cc:348] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.595468 17379 raft_consensus.cc:374] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.595762 17379 raft_consensus.cc:729] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.596462 17379 consensus_queue.cc:260] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.599321 17333 raft_consensus.cc:2454] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 1.
I20240502 01:48:48.600311 17377 tablet_bootstrap.cc:654] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.600598 17147 leader_election.cc:302] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters: b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:48.601467 17390 raft_consensus.cc:686] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:48.601748 17385 raft_consensus.cc:2781] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.603842 17390 consensus_queue.cc:237] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.611594 17379 ts_tablet_manager.cc:1398] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.022s user 0.009s sys 0.000s
I20240502 01:48:48.612746 17379 tablet_bootstrap.cc:492] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.623364 17379 tablet_bootstrap.cc:654] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.631614 17377 tablet_bootstrap.cc:492] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.632370 17377 ts_tablet_manager.cc:1367] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.013s
I20240502 01:48:48.632905 17385 raft_consensus.cc:686] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:48:48.631606 17378 tablet_bootstrap.cc:492] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.633579 17378 ts_tablet_manager.cc:1367] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.016s
I20240502 01:48:48.633769 17385 consensus_queue.cc:237] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.636554 17378 raft_consensus.cc:348] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.637552 17378 raft_consensus.cc:374] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.637970 17378 raft_consensus.cc:729] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.637193 17078 catalog_manager.cc:5484] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 1 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:48.641607 17378 consensus_queue.cc:260] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.642789 17379 tablet_bootstrap.cc:492] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.643308 17379 ts_tablet_manager.cc:1367] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.003s
I20240502 01:48:48.643885 17378 ts_tablet_manager.cc:1398] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:48:48.644855 17378 tablet_bootstrap.cc:492] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.646253 17379 raft_consensus.cc:348] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.647065 17379 raft_consensus.cc:374] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.647418 17379 raft_consensus.cc:729] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.649649 17377 raft_consensus.cc:348] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.652977 17377 raft_consensus.cc:374] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.653290 17377 raft_consensus.cc:729] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.653465 17379 consensus_queue.cc:260] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.653918 17377 consensus_queue.cc:260] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.655855 17377 ts_tablet_manager.cc:1398] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.023s user 0.003s sys 0.002s
I20240502 01:48:48.656359 17379 ts_tablet_manager.cc:1398] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:48:48.656724 17377 tablet_bootstrap.cc:492] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a: Bootstrap starting.
I20240502 01:48:48.659123 17379 tablet_bootstrap.cc:492] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.662684 17377 tablet_bootstrap.cc:654] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.665702 17379 tablet_bootstrap.cc:654] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.666689 17378 tablet_bootstrap.cc:654] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.669315 17077 catalog_manager.cc:5484] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:48.680524 17379 tablet_bootstrap.cc:492] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.681206 17379 ts_tablet_manager.cc:1367] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.003s
I20240502 01:48:48.685552 17379 raft_consensus.cc:348] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.687466 17379 raft_consensus.cc:374] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.687996 17379 raft_consensus.cc:729] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.689095 17378 tablet_bootstrap.cc:492] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.688877 17379 consensus_queue.cc:260] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.689589 17378 ts_tablet_manager.cc:1367] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.016s
I20240502 01:48:48.691718 17379 ts_tablet_manager.cc:1398] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20240502 01:48:48.692839 17379 tablet_bootstrap.cc:492] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.692549 17378 raft_consensus.cc:348] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.693320 17378 raft_consensus.cc:374] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.693631 17378 raft_consensus.cc:729] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.694344 17378 consensus_queue.cc:260] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.698905 17378 ts_tablet_manager.cc:1398] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:48:48.699590 17379 tablet_bootstrap.cc:654] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.699882 17378 tablet_bootstrap.cc:492] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.707432 17378 tablet_bootstrap.cc:654] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.713332 17377 tablet_bootstrap.cc:492] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a: No bootstrap required, opened a new log
I20240502 01:48:48.713840 17377 ts_tablet_manager.cc:1367] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.003s
I20240502 01:48:48.716564 17377 raft_consensus.cc:348] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.717301 17377 raft_consensus.cc:374] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.717577 17377 raft_consensus.cc:729] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.718305 17377 consensus_queue.cc:260] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.720829 17377 ts_tablet_manager.cc:1398] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:48:48.723652 17379 tablet_bootstrap.cc:492] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.724298 17379 ts_tablet_manager.cc:1367] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.015s
I20240502 01:48:48.726980 17379 raft_consensus.cc:348] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.727792 17378 tablet_bootstrap.cc:492] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.727823 17379 raft_consensus.cc:374] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.728430 17378 ts_tablet_manager.cc:1367] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.001s
I20240502 01:48:48.728416 17379 raft_consensus.cc:729] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.729403 17379 consensus_queue.cc:260] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.730614 17385 raft_consensus.cc:482] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.731086 17385 raft_consensus.cc:504] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:48.732532 17379 ts_tablet_manager.cc:1398] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20240502 01:48:48.733129 17385 leader_election.cc:288] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.733690 17379 tablet_bootstrap.cc:492] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.734539 17378 raft_consensus.cc:348] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.735301 17378 raft_consensus.cc:374] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.735610 17378 raft_consensus.cc:729] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.735317 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a" is_pre_election: true
I20240502 01:48:48.736897 17378 consensus_queue.cc:260] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.739502 17378 ts_tablet_manager.cc:1398] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:48:48.740489 17378 tablet_bootstrap.cc:492] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.741070 17379 tablet_bootstrap.cc:654] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.735647 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.744920 17259 raft_consensus.cc:2379] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in current term 1: Already voted for candidate 3c83f4847492417a91fb69cc7afc616a in this term.
I20240502 01:48:48.746147 17148 leader_election.cc:302] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:48.746865 17378 tablet_bootstrap.cc:654] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.747341 17385 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.753716 17379 tablet_bootstrap.cc:492] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.754380 17379 ts_tablet_manager.cc:1367] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20240502 01:48:48.757078 17379 raft_consensus.cc:348] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.757939 17379 raft_consensus.cc:374] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.758262 17379 raft_consensus.cc:729] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.758697 17378 tablet_bootstrap.cc:492] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.759161 17378 ts_tablet_manager.cc:1367] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20240502 01:48:48.758976 17379 consensus_queue.cc:260] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.761961 17379 ts_tablet_manager.cc:1398] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:48:48.762398 17378 raft_consensus.cc:348] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.762841 17379 tablet_bootstrap.cc:492] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3: Bootstrap starting.
I20240502 01:48:48.763070 17378 raft_consensus.cc:374] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.763358 17378 raft_consensus.cc:729] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.764031 17378 consensus_queue.cc:260] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.768417 17378 ts_tablet_manager.cc:1398] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:48:48.768791 17379 tablet_bootstrap.cc:654] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.769343 17378 tablet_bootstrap.cc:492] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0: Bootstrap starting.
I20240502 01:48:48.775569 17378 tablet_bootstrap.cc:654] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:48.786290 17390 raft_consensus.cc:482] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.786818 17378 tablet_bootstrap.cc:492] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0: No bootstrap required, opened a new log
I20240502 01:48:48.787343 17378 ts_tablet_manager.cc:1367] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240502 01:48:48.786784 17390 raft_consensus.cc:504] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.789019 17390 leader_election.cc:288] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.789839 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.789873 17378 raft_consensus.cc:348] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.790760 17378 raft_consensus.cc:374] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.790840 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a" is_pre_election: true
I20240502 01:48:48.791184 17378 raft_consensus.cc:729] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.791855 17333 raft_consensus.cc:2379] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b759976c05b94fdc8be9d54a67b2c1b3 in current term 1: Already voted for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in this term.
I20240502 01:48:48.753928 17385 raft_consensus.cc:2726] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:48.793334 17224 leader_election.cc:302] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b759976c05b94fdc8be9d54a67b2c1b3; no voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:48.793726 17379 tablet_bootstrap.cc:492] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3: No bootstrap required, opened a new log
I20240502 01:48:48.794212 17379 ts_tablet_manager.cc:1367] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.008s
I20240502 01:48:48.794615 17390 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.796700 17379 raft_consensus.cc:348] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.797384 17379 raft_consensus.cc:374] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:48.797638 17379 raft_consensus.cc:729] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Initialized, Role: FOLLOWER
I20240502 01:48:48.798256 17379 consensus_queue.cc:260] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.800201 17379 ts_tablet_manager.cc:1398] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:48:48.800557 17378 consensus_queue.cc:260] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.800940 17390 raft_consensus.cc:2726] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:48.802495 17378 ts_tablet_manager.cc:1398] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20240502 01:48:48.840169 17402 raft_consensus.cc:482] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.840620 17402 raft_consensus.cc:504] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.842289 17402 leader_election.cc:288] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.843158 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097fbd59fa8b49ad85f0b004b3cbcd9d" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.843489 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097fbd59fa8b49ad85f0b004b3cbcd9d" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.843853 17184 raft_consensus.cc:2454] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.844117 17259 raft_consensus.cc:2454] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.844988 17297 leader_election.cc:302] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:48:48.845775 17402 raft_consensus.cc:2781] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.846053 17402 raft_consensus.cc:482] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.846303 17402 raft_consensus.cc:3037] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.850649 17402 raft_consensus.cc:504] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.852207 17402 leader_election.cc:288] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.853139 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097fbd59fa8b49ad85f0b004b3cbcd9d" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:48.853176 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097fbd59fa8b49ad85f0b004b3cbcd9d" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.853682 17184 raft_consensus.cc:3037] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.853854 17259 raft_consensus.cc:3037] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.858300 17184 raft_consensus.cc:2454] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.859248 17297 leader_election.cc:302] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:48:48.859903 17402 raft_consensus.cc:2781] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.860229 17259 raft_consensus.cc:2454] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.860725 17402 raft_consensus.cc:686] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:48.861169 17400 raft_consensus.cc:482] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.861604 17400 raft_consensus.cc:504] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.861701 17402 consensus_queue.cc:237] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.863533 17400 leader_election.cc:288] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.865132 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.865495 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.865859 17259 raft_consensus.cc:2454] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.866091 17390 raft_consensus.cc:482] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.866565 17390 raft_consensus.cc:504] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.866904 17299 leader_election.cc:302] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:48.867709 17400 raft_consensus.cc:2781] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.867988 17184 raft_consensus.cc:2454] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.868103 17400 raft_consensus.cc:482] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.868592 17400 raft_consensus.cc:3037] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.870272 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c83f4847492417a91fb69cc7afc616a" is_pre_election: true
I20240502 01:48:48.871886 17390 leader_election.cc:288] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:48.872738 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.873493 17184 raft_consensus.cc:2379] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b759976c05b94fdc8be9d54a67b2c1b3 in current term 1: Already voted for candidate 3c83f4847492417a91fb69cc7afc616a in this term.
I20240502 01:48:48.874603 17223 leader_election.cc:302] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b759976c05b94fdc8be9d54a67b2c1b3; no voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:48.875260 17400 raft_consensus.cc:504] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.875679 17418 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.874897 17078 catalog_manager.cc:5484] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
W20240502 01:48:48.882624 17126 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:48:48.883615 17400 leader_election.cc:288] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.884912 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:48.885391 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.886030 17259 raft_consensus.cc:3037] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.885457 17184 raft_consensus.cc:3037] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.884929 17418 raft_consensus.cc:2726] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:48.888937 17402 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:48.893020 17184 raft_consensus.cc:2454] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.893904 17259 raft_consensus.cc:2454] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.894080 17297 leader_election.cc:302] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:48:48.895115 17402 raft_consensus.cc:2781] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.895637 17402 raft_consensus.cc:686] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:48.896584 17402 consensus_queue.cc:237] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.901973 17381 raft_consensus.cc:482] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:48.902477 17381 raft_consensus.cc:504] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.904430 17381 leader_election.cc:288] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.904943 17381 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:48.907883 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:48.908505 17259 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.910001 17299 leader_election.cc:302] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:48.911422 17400 raft_consensus.cc:2781] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:48.911798 17400 raft_consensus.cc:482] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:48.912139 17400 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.918762 17400 raft_consensus.cc:504] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.922183 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:48.922852 17184 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:48.924154 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:48.924799 17181 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.929581 17381 consensus_queue.cc:1035] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:48.935205 17181 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.937572 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:48.938254 17259 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.944972 17259 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:48.946120 17299 leader_election.cc:302] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:48.947240 17400 leader_election.cc:288] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:48.948302 17400 raft_consensus.cc:2781] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:48.957476 17400 raft_consensus.cc:686] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:48.959728 17183 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.958472 17400 consensus_queue.cc:237] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:48.968103 17443 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:48.984329 17077 catalog_manager.cc:5484] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:48.984999 17400 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:48.986903 17181 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:48.991894 17400 consensus_queue.cc:1035] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:48.997876 17418 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.006821 17078 catalog_manager.cc:5484] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:49.031448 17390 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.062788 17399 raft_consensus.cc:482] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:49.063210 17399 raft_consensus.cc:504] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.064841 17399 leader_election.cc:288] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:49.065589 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" is_pre_election: true
I20240502 01:48:49.066128 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" is_pre_election: true
I20240502 01:48:49.066335 17181 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:49.066826 17257 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 0.
I20240502 01:48:49.067404 17297 leader_election.cc:302] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:48:49.068220 17399 raft_consensus.cc:2781] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:49.068590 17399 raft_consensus.cc:482] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:49.068929 17399 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:49.075341 17399 raft_consensus.cc:504] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.077418 17399 leader_election.cc:288] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:49.078249 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:49.078894 17181 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:49.081467 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:49.082060 17257 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:49.085211 17181 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:49.086300 17297 leader_election.cc:302] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:48:49.087100 17399 raft_consensus.cc:2781] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:49.087983 17257 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 1.
I20240502 01:48:49.087579 17399 raft_consensus.cc:686] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.089040 17399 consensus_queue.cc:237] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.097047 17077 catalog_manager.cc:5484] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 1 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:49.165036 17421 consensus_queue.cc:1035] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:49.168958 17385 consensus_queue.cc:1035] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:49.205570 17333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:48:49.206148 17333 raft_consensus.cc:595] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Received request to transfer leadership to TS 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:49.211948 17333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:48:49.212491 17333 raft_consensus.cc:595] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Received request to transfer leadership to TS 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:49.228947 17257 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d761bc3026844a54822528ce6e783df8"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:48458
I20240502 01:48:49.229446 17257 raft_consensus.cc:595] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 LEADER]: Received request to transfer leadership to TS 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:49.234457 17333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b067abdcee08465284065c9f6a221414"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:48:49.234946 17333 raft_consensus.cc:595] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:49.239751 17333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:48:49.240273 17333 raft_consensus.cc:595] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:49.245177 17333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:48:49.245680 17333 raft_consensus.cc:595] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:49.250509 17333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55676801f633492291856a8cd6c61819"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:48:49.251209 17333 raft_consensus.cc:595] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:48:49.293440 17399 consensus_queue.cc:1035] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.303205 17441 consensus_queue.cc:1035] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.346678 17441 consensus_queue.cc:1035] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.352025 17441 consensus_queue.cc:1035] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.355517 17400 raft_consensus.cc:979] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.356005 17445 raft_consensus.cc:1067] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.363948 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48450
I20240502 01:48:49.364579 17259 raft_consensus.cc:482] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.364964 17259 raft_consensus.cc:3037] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.372932 17259 raft_consensus.cc:504] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.375015 17259 leader_election.cc:288] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:49.376721 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:49.376897 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ca0cf4837b44489774215a096ac335" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.377350 17183 raft_consensus.cc:3037] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.377516 17333 raft_consensus.cc:3032] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.377918 17333 raft_consensus.cc:729] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.378681 17333 consensus_queue.cc:260] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.380085 17333 raft_consensus.cc:3037] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.384369 17183 raft_consensus.cc:2454] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.385946 17223 leader_election.cc:302] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:49.386828 17390 raft_consensus.cc:2781] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.387303 17390 raft_consensus.cc:686] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:49.388147 17390 consensus_queue.cc:237] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.394105 17399 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.395270 17400 raft_consensus.cc:979] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:48:49.396888 17400 raft_consensus.cc:1067] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Signalling peer 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:48:49.396709 17079 catalog_manager.cc:5484] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 1 to 2, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:49.399014 17182 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:33112
I20240502 01:48:49.399565 17182 raft_consensus.cc:482] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.399933 17182 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.405355 17333 raft_consensus.cc:2454] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.407900 17182 raft_consensus.cc:504] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:49.412425 17258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:49.413080 17258 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.413461 17182 leader_election.cc:288] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:49.412954 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.415370 17333 raft_consensus.cc:3032] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.415788 17333 raft_consensus.cc:729] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.416750 17333 consensus_queue.cc:260] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:49.417678 17400 raft_consensus.cc:979] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.418216 17400 raft_consensus.cc:1067] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.418972 17333 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.420059 17257 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b067abdcee08465284065c9f6a221414"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48450
I20240502 01:48:49.420629 17257 raft_consensus.cc:482] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.421002 17257 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.424814 17399 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.427103 17400 raft_consensus.cc:979] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.427529 17400 raft_consensus.cc:1067] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.430317 17256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48450
I20240502 01:48:49.430975 17256 raft_consensus.cc:482] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.431361 17256 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.438894 17257 raft_consensus.cc:504] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.445468 17258 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 2.
I20240502 01:48:49.447029 17148 leader_election.cc:302] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:49.448103 17421 raft_consensus.cc:2781] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.448252 17259 raft_consensus.cc:1226] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Rejecting Update request from peer 3c83f4847492417a91fb69cc7afc616a for earlier term 1. Current term is 2. Ops: []
I20240502 01:48:49.448706 17421 raft_consensus.cc:686] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:48:49.449225 17257 leader_election.cc:288] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:49.449712 17421 consensus_queue.cc:237] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:48:49.452659 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:49.454138 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.454025 17399 consensus_queue.cc:1046] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 01:48:49.457968 17332 raft_consensus.cc:3032] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.458312 17332 raft_consensus.cc:729] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.460019 17333 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 2.
I20240502 01:48:49.461251 17332 consensus_queue.cc:260] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.462453 17256 raft_consensus.cc:504] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.463855 17332 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.465211 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:49.465816 17182 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.472832 17182 raft_consensus.cc:2454] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.472908 17079 catalog_manager.cc:5484] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 1 to 2, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 2 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:49.473989 17223 leader_election.cc:302] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:49.474956 17390 raft_consensus.cc:2781] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.475443 17390 raft_consensus.cc:686] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:49.476372 17390 consensus_queue.cc:237] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.477857 17445 raft_consensus.cc:979] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:48:49.478273 17400 raft_consensus.cc:1067] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Signalling peer 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:48:49.482352 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.483119 17333 raft_consensus.cc:3032] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.483506 17333 raft_consensus.cc:729] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.484442 17333 consensus_queue.cc:260] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.485430 17181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:33112
I20240502 01:48:49.486490 17181 raft_consensus.cc:482] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.476810 17332 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.485845 17256 leader_election.cc:288] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
W20240502 01:48:49.487269 17399 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240502 01:48:49.487027 17181 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.489398 17399 consensus_queue.cc:1035] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.486552 17333 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.484966 17078 catalog_manager.cc:5484] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 1 to 2, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:49.491536 17224 leader_election.cc:302] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters: 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:48:49.492785 17418 raft_consensus.cc:2781] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.493284 17418 raft_consensus.cc:686] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:49.494113 17418 consensus_queue.cc:237] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.498121 17333 raft_consensus.cc:2454] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.503998 17181 raft_consensus.cc:504] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.506691 17077 catalog_manager.cc:5484] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 1 to 2, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:49.508610 17256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:49.509224 17256 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.520567 17399 consensus_queue.cc:1035] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:49.527387 17256 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 2.
I20240502 01:48:49.531479 17148 leader_election.cc:302] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:49.532562 17421 raft_consensus.cc:2781] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.533218 17421 raft_consensus.cc:686] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:48:49.534968 17181 leader_election.cc:288] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 2 election: Requested vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:49.534776 17421 consensus_queue.cc:237] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.538437 17399 raft_consensus.cc:979] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.538861 17400 raft_consensus.cc:1067] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:48:49.540858 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55676801f633492291856a8cd6c61819"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48450
I20240502 01:48:49.541651 17259 raft_consensus.cc:482] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.542146 17259 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.543423 17075 catalog_manager.cc:5484] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 1 to 2, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 2 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:49.547019 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.547935 17333 raft_consensus.cc:3032] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.548281 17333 raft_consensus.cc:729] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.549074 17259 raft_consensus.cc:504] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.549170 17333 consensus_queue.cc:260] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.551702 17259 leader_election.cc:288] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:49.552820 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.553512 17332 raft_consensus.cc:3032] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.553638 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:49.553926 17332 raft_consensus.cc:729] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:49.554361 17181 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.554988 17332 consensus_queue.cc:260] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.556576 17332 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.562011 17181 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.563329 17223 leader_election.cc:302] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:49.564334 17418 raft_consensus.cc:2781] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.564841 17418 raft_consensus.cc:686] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:49.565706 17418 consensus_queue.cc:237] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.574165 17390 raft_consensus.cc:979] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:48:49.573472 17333 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.574658 17390 raft_consensus.cc:1067] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 LEADER]: Signalling peer 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:48:49.576891 17183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bc3026844a54822528ce6e783df8"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:33120
I20240502 01:48:49.577322 17332 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 2.
I20240502 01:48:49.577719 17183 raft_consensus.cc:482] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:49.578231 17183 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.576886 17075 catalog_manager.cc:5484] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 1 to 2, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:49.590867 17183 raft_consensus.cc:504] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.592586 17333 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 2.
I20240502 01:48:49.593657 17183 leader_election.cc:288] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 2 election: Requested vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:48:49.593905 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:49.594584 17259 raft_consensus.cc:3032] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:49.594924 17259 raft_consensus.cc:729] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:49.595315 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:48:49.595698 17259 consensus_queue.cc:260] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.595824 17332 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.596765 17259 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.602082 17259 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 2.
I20240502 01:48:49.603390 17148 leader_election.cc:302] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:49.604230 17332 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 2.
I20240502 01:48:49.604302 17421 raft_consensus.cc:2781] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:49.604898 17421 raft_consensus.cc:686] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:48:49.605782 17331 raft_consensus.cc:1226] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Rejecting Update request from peer b759976c05b94fdc8be9d54a67b2c1b3 for earlier term 1. Current term is 2. Ops: []
I20240502 01:48:49.605787 17421 consensus_queue.cc:237] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:49.613233 17075 catalog_manager.cc:5484] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 1 to 2, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 2 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:49.890740 17331 raft_consensus.cc:1261] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.892292 17468 consensus_queue.cc:1035] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.900336 17183 raft_consensus.cc:1261] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.901988 17468 consensus_queue.cc:1035] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.903324 17259 raft_consensus.cc:1261] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.906599 17421 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.914724 17331 raft_consensus.cc:1261] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.916241 17421 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.928426 17259 raft_consensus.cc:1261] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.929737 17385 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.937723 17331 raft_consensus.cc:1261] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.938848 17469 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.968948 17331 raft_consensus.cc:1261] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.970095 17468 consensus_queue.cc:1035] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.977494 17183 raft_consensus.cc:1261] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.979450 17390 consensus_queue.cc:1035] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.983282 17331 raft_consensus.cc:1261] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:49.986470 17390 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:49.993788 17183 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:49.999255 17183 raft_consensus.cc:1261] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:50.000581 17390 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:50.004745 17259 raft_consensus.cc:1261] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:50.006177 17385 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:50.014348 17331 raft_consensus.cc:1261] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:50.016937 17421 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:50.049213 17331 raft_consensus.cc:1261] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:50.050441 17468 consensus_queue.cc:1035] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:50.057649 17183 raft_consensus.cc:1261] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:50.059168 17418 consensus_queue.cc:1035] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:52.282565 17046 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240502 01:48:52.283325 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:52.304179 17183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
mode: GRACEFUL
new_leader_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:33132
I20240502 01:48:52.304672 17183 raft_consensus.cc:595] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 LEADER]: Received request to transfer leadership to TS 3c83f4847492417a91fb69cc7afc616a
I20240502 01:48:52.314539 17256 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b067abdcee08465284065c9f6a221414"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:48464
I20240502 01:48:52.315121 17256 raft_consensus.cc:595] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Received request to transfer leadership to TS 3c83f4847492417a91fb69cc7afc616a
I20240502 01:48:52.316251 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 01:48:52.316655 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:52.331171 17390 raft_consensus.cc:979] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:48:52.331583 17475 raft_consensus.cc:1067] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Signalling peer 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:48:52.332953 17331 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b067abdcee08465284065c9f6a221414"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:45266
I20240502 01:48:52.333390 17331 raft_consensus.cc:482] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:52.333647 17331 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:48:52.338711 17331 raft_consensus.cc:504] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:52.340284 17331 leader_election.cc:288] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 3 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:52.341009 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:52.341364 17256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:52.341509 17183 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:48:52.341847 17256 raft_consensus.cc:3032] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:48:52.342120 17256 raft_consensus.cc:729] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:48:52.342746 17256 consensus_queue.cc:260] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:52.343626 17256 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:48:52.360754 17256 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 3.
I20240502 01:48:52.360752 17183 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 3.
I20240502 01:48:52.362124 17299 leader_election.cc:302] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:52.362823 17470 raft_consensus.cc:2781] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:48:52.363940 17470 raft_consensus.cc:686] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:52.364594 17470 consensus_queue.cc:237] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:52.370676 17079 catalog_manager.cc:5484] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 2 to 3, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 3 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:52.430178 17471 raft_consensus.cc:979] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: : Instructing follower 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:48:52.430588 17421 raft_consensus.cc:1067] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 LEADER]: Signalling peer 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:48:52.431897 17331 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:45264
I20240502 01:48:52.432363 17331 raft_consensus.cc:482] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:48:52.432636 17331 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:52.436859 17331 raft_consensus.cc:504] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:52.438441 17331 leader_election.cc:288] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 2 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:48:52.439200 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:48:52.439652 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:48:52.439802 17183 raft_consensus.cc:3032] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:48:52.440258 17259 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:52.440227 17183 raft_consensus.cc:729] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:48:52.441176 17183 consensus_queue.cc:260] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:52.442147 17183 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:52.444690 17259 raft_consensus.cc:2454] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 2.
I20240502 01:48:52.445606 17299 leader_election.cc:302] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:48:52.446238 17470 raft_consensus.cc:2781] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:48:52.446560 17183 raft_consensus.cc:2454] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 2.
I20240502 01:48:52.446712 17470 raft_consensus.cc:686] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:48:52.447569 17470 consensus_queue.cc:237] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:48:52.453441 17079 catalog_manager.cc:5484] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 1 to 2, leader changed from 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129) to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 2 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:52.758852 17183 raft_consensus.cc:1261] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:48:52.760051 17470 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:48:52.772876 17259 raft_consensus.cc:1261] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:48:52.774988 17470 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:48:52.921597 17259 raft_consensus.cc:1261] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:52.922770 17470 consensus_queue.cc:1035] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:52.931281 17183 raft_consensus.cc:1261] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:48:52.933449 17489 consensus_queue.cc:1035] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:48:53.317409 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:53.322217 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:53.322643 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:54.323487 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:54.328440 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:54.328895 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:55.329756 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:55.334594 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:55.335057 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:56.336108 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:56.341163 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:56.341598 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:57.342422 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:57.347319 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:57.347757 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:58.348628 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:58.354012 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:58.354475 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:59.355408 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:59.360273 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:59.360754 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:00.361672 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:00.367070 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:00.367552 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:01.368420 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:01.373246 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:01.373679 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:02.398217 17046 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240502 01:49:02.418254 17181 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:46558
I20240502 01:49:02.418964 17181 raft_consensus.cc:595] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:49:02.424440 17181 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:46558
I20240502 01:49:02.424942 17181 raft_consensus.cc:595] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:49:02.429481 17331 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:49:02.429977 17331 raft_consensus.cc:595] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:49:02.434779 17181 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d761bc3026844a54822528ce6e783df8"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:46558
I20240502 01:49:02.435258 17181 raft_consensus.cc:595] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:49:02.439637 17331 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b067abdcee08465284065c9f6a221414"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
mode: GRACEFUL
new_leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:45276
I20240502 01:49:02.440156 17331 raft_consensus.cc:595] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Received request to transfer leadership to TS b759976c05b94fdc8be9d54a67b2c1b3
I20240502 01:49:02.484576 17500 raft_consensus.cc:979] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.485148 17494 raft_consensus.cc:1067] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.486742 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b067abdcee08465284065c9f6a221414"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48450
I20240502 01:49:02.487207 17259 raft_consensus.cc:482] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:02.487565 17259 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:02.492484 17259 raft_consensus.cc:504] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.494210 17259 leader_election.cc:288] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 4 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:49:02.495123 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:02.495322 17331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b067abdcee08465284065c9f6a221414" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:02.495950 17181 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:02.496011 17331 raft_consensus.cc:3032] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:49:02.496409 17331 raft_consensus.cc:729] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:49:02.497221 17331 consensus_queue.cc:260] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.498606 17331 raft_consensus.cc:3037] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:02.502507 17181 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 4.
I20240502 01:49:02.503952 17223 leader_election.cc:302] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:49:02.504951 17495 raft_consensus.cc:2781] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:49:02.505262 17331 raft_consensus.cc:2454] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 4.
I20240502 01:49:02.508214 17495 raft_consensus.cc:686] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:02.508968 17495 consensus_queue.cc:237] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.516858 17496 raft_consensus.cc:979] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.517506 17491 raft_consensus.cc:1067] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.516835 17079 catalog_manager.cc:5484] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 3 to 4, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 4 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.519012 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48438
I20240502 01:49:02.519491 17259 raft_consensus.cc:482] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:02.519789 17259 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.525811 17259 raft_consensus.cc:504] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:49:02.527504 17259 leader_election.cc:288] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Requested vote from peers 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061), 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677)
I20240502 01:49:02.528442 17331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:02.528609 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:02.529117 17331 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.529229 17181 raft_consensus.cc:3032] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:02.529605 17181 raft_consensus.cc:729] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:49:02.530398 17181 consensus_queue.cc:260] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:49:02.531755 17181 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.535009 17331 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.536108 17491 raft_consensus.cc:979] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.536171 17224 leader_election.cc:302] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:49:02.536657 17496 raft_consensus.cc:1067] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.537061 17495 raft_consensus.cc:2781] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:02.537547 17495 raft_consensus.cc:686] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:02.537876 17181 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.538146 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bc3026844a54822528ce6e783df8"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48438
I20240502 01:49:02.538575 17259 raft_consensus.cc:482] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:02.538919 17259 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.538404 17495 consensus_queue.cc:237] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:49:02.543613 17259 raft_consensus.cc:504] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.544957 17259 leader_election.cc:288] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:49:02.545830 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:02.546115 17331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:02.545506 17079 catalog_manager.cc:5484] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 2 to 3, leader changed from 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 3 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.546381 17181 raft_consensus.cc:3032] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:02.546710 17181 raft_consensus.cc:729] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:49:02.547245 17181 consensus_queue.cc:260] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.548180 17331 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.548240 17181 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.553558 17181 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.554280 17331 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.554589 17223 leader_election.cc:302] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:49:02.555351 17495 raft_consensus.cc:2781] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:02.555738 17495 raft_consensus.cc:686] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:02.556527 17495 consensus_queue.cc:237] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.562947 17079 catalog_manager.cc:5484] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 2 to 3, leader changed from 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 3 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.715298 17500 raft_consensus.cc:979] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.715735 17494 raft_consensus.cc:1067] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.717116 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48450
I20240502 01:49:02.717568 17259 raft_consensus.cc:482] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:02.717833 17259 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.721652 17259 raft_consensus.cc:504] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.723069 17259 leader_election.cc:288] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:49:02.723879 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:02.724103 17331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5260c565a7ea4e06ab2c9ea242e287f4" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:02.724332 17181 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.724593 17331 raft_consensus.cc:3032] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:02.724861 17331 raft_consensus.cc:729] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:49:02.725514 17331 consensus_queue.cc:260] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.726392 17331 raft_consensus.cc:3037] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.728516 17181 raft_consensus.cc:2454] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.729437 17223 leader_election.cc:302] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa7b2d1ea57421ba758a4231c9bcaf0, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:49:02.730252 17495 raft_consensus.cc:2781] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:02.730623 17495 raft_consensus.cc:686] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:02.731282 17331 raft_consensus.cc:2454] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.731324 17495 consensus_queue.cc:237] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.738219 17079 catalog_manager.cc:5484] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 2 to 3, leader changed from 3c83f4847492417a91fb69cc7afc616a (127.16.165.131) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 3 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.797200 17496 raft_consensus.cc:979] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: : Instructing follower b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.797721 17496 raft_consensus.cc:1067] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Signalling peer b759976c05b94fdc8be9d54a67b2c1b3 to start an election
I20240502 01:49:02.799111 17259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
from {username='slave'} at 127.0.0.1:48438
I20240502 01:49:02.799522 17259 raft_consensus.cc:482] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:02.799748 17259 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.803666 17259 raft_consensus.cc:504] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.805117 17259 leader_election.cc:288] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:49:02.806005 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:02.806214 17331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:02.806468 17181 raft_consensus.cc:3032] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:02.806740 17181 raft_consensus.cc:729] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:49:02.806759 17331 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.807349 17181 consensus_queue.cc:260] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.808306 17181 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:02.811393 17331 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.812678 17224 leader_election.cc:302] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, b759976c05b94fdc8be9d54a67b2c1b3; no voters:
I20240502 01:49:02.812681 17181 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b759976c05b94fdc8be9d54a67b2c1b3 in term 3.
I20240502 01:49:02.813628 17495 raft_consensus.cc:2781] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:02.814074 17495 raft_consensus.cc:686] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Leader. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:02.814882 17495 consensus_queue.cc:237] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:02.822002 17079 catalog_manager.cc:5484] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 reported cstate change: term changed from 2 to 3, leader changed from 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129) to b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130). New cstate: current_term: 3 leader_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.924577 17181 raft_consensus.cc:1261] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:02.925971 17495 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:02.935918 17331 raft_consensus.cc:1261] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:02.937431 17495 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:02.974560 17181 raft_consensus.cc:1261] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:02.975798 17495 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:02.984390 17331 raft_consensus.cc:1261] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:02.986115 17495 consensus_queue.cc:1035] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:03.001724 17183 raft_consensus.cc:1261] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:03.003257 17501 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:03.012398 17331 raft_consensus.cc:1261] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:03.014400 17501 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:03.226825 17181 raft_consensus.cc:1261] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:03.227945 17501 consensus_queue.cc:1035] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:03.237265 17331 raft_consensus.cc:1261] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:03.238377 17501 consensus_queue.cc:1035] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:03.255569 17181 raft_consensus.cc:1261] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:03.256824 17501 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:03.265731 17331 raft_consensus.cc:1261] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Refusing update from remote peer b759976c05b94fdc8be9d54a67b2c1b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:03.270028 17495 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:05.474560 17046 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240502 01:49:05.475250 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:05.481877 17259 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55676801f633492291856a8cd6c61819"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:48464
I20240502 01:49:05.482358 17259 raft_consensus.cc:595] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Received request to transfer leadership to TS 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:49:05.484774 17259 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:48464
I20240502 01:49:05.485266 17259 raft_consensus.cc:595] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Received request to transfer leadership to TS 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:49:05.487361 17259 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:48464
I20240502 01:49:05.487840 17259 raft_consensus.cc:595] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Received request to transfer leadership to TS 6fa7b2d1ea57421ba758a4231c9bcaf0
I20240502 01:49:05.489905 17259 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d761bc3026844a54822528ce6e783df8"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:48464
I20240502 01:49:05.490336 17259 raft_consensus.cc:595] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Received request to transfer leadership to TS 3c83f4847492417a91fb69cc7afc616a
I20240502 01:49:05.492535 17259 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73"
dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
mode: GRACEFUL
new_leader_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:48464
I20240502 01:49:05.493083 17259 raft_consensus.cc:595] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Received request to transfer leadership to TS 3c83f4847492417a91fb69cc7afc616a
I20240502 01:49:05.508798 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 01:49:05.513834 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:05.518862 17495 raft_consensus.cc:979] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:49:05.519466 17530 raft_consensus.cc:1067] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Signalling peer 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:49:05.521135 17181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:33120
I20240502 01:49:05.521729 17181 raft_consensus.cc:482] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:05.522070 17181 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.530469 17181 raft_consensus.cc:504] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:49:05.532788 17181 leader_election.cc:288] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:49:05.534586 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:49:05.534611 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8576363f2f44f7b349932695b7fa58" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:05.535344 17259 raft_consensus.cc:3032] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:49:05.535451 17332 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.535631 17259 raft_consensus.cc:729] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:05.536396 17259 consensus_queue.cc:260] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:49:05.537526 17259 raft_consensus.cc:3037] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.540676 17332 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 4.
I20240502 01:49:05.542083 17147 leader_election.cc:302] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:49:05.542409 17259 raft_consensus.cc:2454] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 4.
I20240502 01:49:05.543809 17531 raft_consensus.cc:2781] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:49:05.544625 17531 raft_consensus.cc:686] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:49:05.545665 17531 consensus_queue.cc:237] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } }
I20240502 01:49:05.553556 17075 catalog_manager.cc:5484] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 3 to 4, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 4 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:05.607292 17495 raft_consensus.cc:979] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:49:05.607806 17530 raft_consensus.cc:1067] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Signalling peer 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:49:05.609441 17332 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:45266
I20240502 01:49:05.609995 17332 raft_consensus.cc:482] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:05.610306 17332 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:05.615096 17332 raft_consensus.cc:504] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.617311 17332 leader_election.cc:288] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 3 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:49:05.618060 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:05.618503 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7738ff80ca94c59bfe1bcd5f3c0bc73" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:49:05.618618 17181 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:05.618968 17259 raft_consensus.cc:3032] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:05.619189 17259 raft_consensus.cc:729] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:05.619764 17259 consensus_queue.cc:260] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.620788 17259 raft_consensus.cc:3037] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:05.623039 17181 raft_consensus.cc:2454] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 3.
I20240502 01:49:05.624235 17297 leader_election.cc:302] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:49:05.625228 17494 raft_consensus.cc:2781] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:05.625361 17259 raft_consensus.cc:2454] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 3.
I20240502 01:49:05.626482 17494 raft_consensus.cc:686] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:49:05.627455 17494 consensus_queue.cc:237] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.634441 17075 catalog_manager.cc:5484] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 2 to 3, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 3 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:05.823504 17530 raft_consensus.cc:979] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:49:05.824043 17495 raft_consensus.cc:1067] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Signalling peer 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:49:05.825716 17181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:33120
I20240502 01:49:05.826182 17181 raft_consensus.cc:482] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:05.826471 17181 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.830883 17181 raft_consensus.cc:504] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.832293 17181 leader_election.cc:288] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 4 election: Requested vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:49:05.833191 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:49:05.833262 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e78e9fa9e74827839975255f3f4d3b" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:05.833758 17259 raft_consensus.cc:3032] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:49:05.833791 17332 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.834075 17259 raft_consensus.cc:729] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:05.834674 17259 consensus_queue.cc:260] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.835572 17259 raft_consensus.cc:3037] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.838084 17332 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 4.
I20240502 01:49:05.838915 17147 leader_election.cc:302] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:49:05.839525 17531 raft_consensus.cc:2781] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:49:05.839691 17259 raft_consensus.cc:2454] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 4.
I20240502 01:49:05.839941 17531 raft_consensus.cc:686] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:49:05.840777 17531 consensus_queue.cc:237] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.847515 17075 catalog_manager.cc:5484] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 3 to 4, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 4 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:05.911090 17530 raft_consensus.cc:979] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:49:05.911556 17495 raft_consensus.cc:1067] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Signalling peer 3c83f4847492417a91fb69cc7afc616a to start an election
I20240502 01:49:05.912920 17332 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bc3026844a54822528ce6e783df8"
dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
from {username='slave'} at 127.0.0.1:45266
I20240502 01:49:05.913314 17332 raft_consensus.cc:482] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:05.913529 17332 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.917716 17332 raft_consensus.cc:504] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.919140 17332 leader_election.cc:288] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 4 election: Requested vote from peers 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677), b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481)
I20240502 01:49:05.919919 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
I20240502 01:49:05.920161 17259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bc3026844a54822528ce6e783df8" candidate_uuid: "3c83f4847492417a91fb69cc7afc616a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:49:05.920445 17181 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.920662 17259 raft_consensus.cc:3032] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:49:05.920929 17259 raft_consensus.cc:729] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:05.921466 17259 consensus_queue.cc:260] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.922425 17259 raft_consensus.cc:3037] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:05.924796 17181 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 4.
I20240502 01:49:05.925683 17297 leader_election.cc:302] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:49:05.926424 17494 raft_consensus.cc:2781] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:49:05.926853 17494 raft_consensus.cc:686] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 4 LEADER]: Becoming Leader. State: Replica: 3c83f4847492417a91fb69cc7afc616a, State: Running, Role: LEADER
I20240502 01:49:05.927243 17259 raft_consensus.cc:2454] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c83f4847492417a91fb69cc7afc616a in term 4.
I20240502 01:49:05.928117 17256 raft_consensus.cc:1261] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:05.927738 17494 consensus_queue.cc:237] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.929347 17531 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:05.935664 17075 catalog_manager.cc:5484] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a reported cstate change: term changed from 3 to 4, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 3c83f4847492417a91fb69cc7afc616a (127.16.165.131). New cstate: current_term: 4 leader_uuid: "3c83f4847492417a91fb69cc7afc616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:05.943229 17332 raft_consensus.cc:1261] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:05.945156 17531 consensus_queue.cc:1035] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:05.984705 17530 raft_consensus.cc:979] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: : Instructing follower 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:49:05.985168 17530 raft_consensus.cc:1067] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Signalling peer 6fa7b2d1ea57421ba758a4231c9bcaf0 to start an election
I20240502 01:49:05.986527 17181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55676801f633492291856a8cd6c61819"
dest_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0"
from {username='slave'} at 127.0.0.1:33120
I20240502 01:49:05.986970 17181 raft_consensus.cc:482] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:05.987241 17181 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:05.991338 17181 raft_consensus.cc:504] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.992738 17181 leader_election.cc:288] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 3 election: Requested vote from peers b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130:44481), 3c83f4847492417a91fb69cc7afc616a (127.16.165.131:36061)
I20240502 01:49:05.993680 17256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b759976c05b94fdc8be9d54a67b2c1b3"
I20240502 01:49:05.993762 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55676801f633492291856a8cd6c61819" candidate_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c83f4847492417a91fb69cc7afc616a"
I20240502 01:49:05.994282 17256 raft_consensus.cc:3032] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:05.994341 17332 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:05.994565 17256 raft_consensus.cc:729] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b759976c05b94fdc8be9d54a67b2c1b3, State: Running, Role: LEADER
I20240502 01:49:05.995237 17256 consensus_queue.cc:260] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:05.996312 17256 raft_consensus.cc:3037] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:05.998754 17332 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 3.
I20240502 01:49:05.999742 17147 leader_election.cc:302] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c83f4847492417a91fb69cc7afc616a, 6fa7b2d1ea57421ba758a4231c9bcaf0; no voters:
I20240502 01:49:06.000556 17531 raft_consensus.cc:2781] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:06.000934 17531 raft_consensus.cc:686] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 6fa7b2d1ea57421ba758a4231c9bcaf0, State: Running, Role: LEADER
I20240502 01:49:06.001242 17256 raft_consensus.cc:2454] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa7b2d1ea57421ba758a4231c9bcaf0 in term 3.
I20240502 01:49:06.001729 17531 consensus_queue.cc:237] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } }
I20240502 01:49:06.008203 17079 catalog_manager.cc:5484] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 reported cstate change: term changed from 2 to 3, leader changed from b759976c05b94fdc8be9d54a67b2c1b3 (127.16.165.130) to 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129). New cstate: current_term: 3 leader_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:06.127259 17181 raft_consensus.cc:1261] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:06.128480 17494 consensus_queue.cc:1035] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:06.137565 17256 raft_consensus.cc:1261] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:06.139649 17539 consensus_queue.cc:1035] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:06.245244 17332 raft_consensus.cc:1261] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:06.246363 17531 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:06.254637 17256 raft_consensus.cc:1261] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:06.256318 17554 consensus_queue.cc:1035] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:49:06.357961 17256 raft_consensus.cc:1261] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:06.359184 17539 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:06.368119 17181 raft_consensus.cc:1261] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Refusing update from remote peer 3c83f4847492417a91fb69cc7afc616a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:06.370402 17557 consensus_queue.cc:1035] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa7b2d1ea57421ba758a4231c9bcaf0" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:49:06.445320 17332 raft_consensus.cc:1261] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:06.446375 17554 consensus_queue.cc:1035] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c83f4847492417a91fb69cc7afc616a" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:06.454713 17256 raft_consensus.cc:1261] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Refusing update from remote peer 6fa7b2d1ea57421ba758a4231c9bcaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:06.456002 17554 consensus_queue.cc:1035] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b759976c05b94fdc8be9d54a67b2c1b3" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:06.515311 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:06.520243 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:06.520671 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:07.521380 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:07.526050 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:07.526484 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:08.527314 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:08.533262 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:08.533674 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:09.534458 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:09.539263 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:09.539664 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:10.540467 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:10.545277 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:10.545691 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:11.546489 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:11.551221 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:11.551640 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:12.686451 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:12.691121 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:12.691511 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:49:12.907353 17280 debug-util.cc:398] Leaking SignalData structure 0x7b0800133060 after lost signal to thread 17049
W20240502 01:49:12.908152 17280 debug-util.cc:398] Leaking SignalData structure 0x7b0800133740 after lost signal to thread 17113
W20240502 01:49:12.908911 17280 debug-util.cc:398] Leaking SignalData structure 0x7b08001ffc80 after lost signal to thread 17208
W20240502 01:49:12.909576 17280 debug-util.cc:398] Leaking SignalData structure 0x7b08001c9280 after lost signal to thread 17283
W20240502 01:49:12.910279 17280 debug-util.cc:398] Leaking SignalData structure 0x7b0800193400 after lost signal to thread 17358
I20240502 01:49:13.692307 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:13.697258 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:13.697705 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:14.698508 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:14.703333 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:14.703781 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:15.729296 17046 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240502 01:49:15.730000 17046 tablet_server.cc:178] TabletServer@127.16.165.129:0 shutting down...
I20240502 01:49:15.758026 17046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:15.758731 17046 tablet_replica.cc:330] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.759560 17046 raft_consensus.cc:2229] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:15.760776 17046 raft_consensus.cc:2258] T 55676801f633492291856a8cd6c61819 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.763219 17046 tablet_replica.cc:330] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.763761 17046 raft_consensus.cc:2229] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.764331 17046 raft_consensus.cc:2258] T d761bc3026844a54822528ce6e783df8 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.765996 17046 tablet_replica.cc:330] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.766501 17046 raft_consensus.cc:2229] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.767025 17046 raft_consensus.cc:2258] T b067abdcee08465284065c9f6a221414 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.769136 17046 tablet_replica.cc:330] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.769721 17046 raft_consensus.cc:2229] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 LEADER]: Raft consensus shutting down.
I20240502 01:49:15.770774 17046 raft_consensus.cc:2258] T 8e8576363f2f44f7b349932695b7fa58 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.772758 17046 tablet_replica.cc:330] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.773255 17046 raft_consensus.cc:2229] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 LEADER]: Raft consensus shutting down.
I20240502 01:49:15.774160 17046 raft_consensus.cc:2258] T 98e78e9fa9e74827839975255f3f4d3b P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.776151 17046 tablet_replica.cc:330] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.776849 17046 raft_consensus.cc:2229] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.777388 17046 raft_consensus.cc:2258] T b0ca0cf4837b44489774215a096ac335 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.779476 17046 tablet_replica.cc:330] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.780160 17046 raft_consensus.cc:2229] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.780758 17046 raft_consensus.cc:2258] T 5260c565a7ea4e06ab2c9ea242e287f4 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.782877 17046 tablet_replica.cc:330] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.783506 17046 raft_consensus.cc:2229] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.784126 17046 raft_consensus.cc:2258] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.786267 17046 tablet_replica.cc:330] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0: stopping tablet replica
I20240502 01:49:15.786875 17046 raft_consensus.cc:2229] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.787456 17046 raft_consensus.cc:2258] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 6fa7b2d1ea57421ba758a4231c9bcaf0 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 01:49:15.898068 17297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.165.129:44677: connect: Connection refused (error 111)
I20240502 01:49:15.901006 17046 tablet_server.cc:194] TabletServer@127.16.165.129:0 shutdown complete.
W20240502 01:49:15.904428 17297 consensus_peers.cc:485] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a -> Peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Couldn't send request to peer 6fa7b2d1ea57421ba758a4231c9bcaf0. Status: Network error: Client connection negotiation failed: client connection to 127.16.165.129:44677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:49:15.904897 17297 consensus_peers.cc:485] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a -> Peer 6fa7b2d1ea57421ba758a4231c9bcaf0 (127.16.165.129:44677): Couldn't send request to peer 6fa7b2d1ea57421ba758a4231c9bcaf0. Status: Network error: Client connection negotiation failed: client connection to 127.16.165.129:44677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:15.920869 17046 tablet_server.cc:178] TabletServer@127.16.165.130:0 shutting down...
I20240502 01:49:15.944151 17046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:15.944849 17046 tablet_replica.cc:330] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.945426 17046 raft_consensus.cc:2229] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:49:15.946314 17046 raft_consensus.cc:2258] T b0ca0cf4837b44489774215a096ac335 P b759976c05b94fdc8be9d54a67b2c1b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.948105 17046 tablet_replica.cc:330] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.948658 17046 raft_consensus.cc:2229] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:15.949450 17046 raft_consensus.cc:2258] T 5260c565a7ea4e06ab2c9ea242e287f4 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.951146 17046 tablet_replica.cc:330] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.951671 17046 raft_consensus.cc:2229] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.952205 17046 raft_consensus.cc:2258] T 55676801f633492291856a8cd6c61819 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.953838 17046 tablet_replica.cc:330] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.954334 17046 raft_consensus.cc:2229] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.954727 17046 raft_consensus.cc:2258] T 097fbd59fa8b49ad85f0b004b3cbcd9d P b759976c05b94fdc8be9d54a67b2c1b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.956640 17046 tablet_replica.cc:330] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.957085 17046 raft_consensus.cc:2229] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.957466 17046 raft_consensus.cc:2258] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P b759976c05b94fdc8be9d54a67b2c1b3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.959049 17046 tablet_replica.cc:330] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.959508 17046 raft_consensus.cc:2229] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 LEADER]: Raft consensus shutting down.
I20240502 01:49:15.960477 17046 raft_consensus.cc:2258] T b067abdcee08465284065c9f6a221414 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.962098 17046 tablet_replica.cc:330] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.962556 17046 raft_consensus.cc:2229] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.962967 17046 raft_consensus.cc:2258] T 8e8576363f2f44f7b349932695b7fa58 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.964767 17046 tablet_replica.cc:330] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.965178 17046 raft_consensus.cc:2229] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.965605 17046 raft_consensus.cc:2258] T 98e78e9fa9e74827839975255f3f4d3b P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:15.967245 17046 tablet_replica.cc:330] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3: stopping tablet replica
I20240502 01:49:15.967643 17046 raft_consensus.cc:2229] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:15.968106 17046 raft_consensus.cc:2258] T d761bc3026844a54822528ce6e783df8 P b759976c05b94fdc8be9d54a67b2c1b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.010139 17046 tablet_server.cc:194] TabletServer@127.16.165.130:0 shutdown complete.
I20240502 01:49:16.025889 17046 tablet_server.cc:178] TabletServer@127.16.165.131:0 shutting down...
I20240502 01:49:16.052667 17046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:16.053368 17046 tablet_replica.cc:330] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.054004 17046 raft_consensus.cc:2229] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.054605 17046 raft_consensus.cc:2258] T 55676801f633492291856a8cd6c61819 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.056524 17046 tablet_replica.cc:330] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.057081 17046 raft_consensus.cc:2229] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.057555 17046 raft_consensus.cc:2258] T b067abdcee08465284065c9f6a221414 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.059212 17046 tablet_replica.cc:330] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.059724 17046 raft_consensus.cc:2229] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:16.060576 17046 raft_consensus.cc:2258] T 097fbd59fa8b49ad85f0b004b3cbcd9d P 3c83f4847492417a91fb69cc7afc616a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.062247 17046 tablet_replica.cc:330] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.062752 17046 raft_consensus.cc:2229] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:16.063591 17046 raft_consensus.cc:2258] T e7738ff80ca94c59bfe1bcd5f3c0bc73 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.065379 17046 tablet_replica.cc:330] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.065856 17046 raft_consensus.cc:2229] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.066256 17046 raft_consensus.cc:2258] T b0ca0cf4837b44489774215a096ac335 P 3c83f4847492417a91fb69cc7afc616a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.068037 17046 tablet_replica.cc:330] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.068501 17046 raft_consensus.cc:2229] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.068925 17046 raft_consensus.cc:2258] T 5260c565a7ea4e06ab2c9ea242e287f4 P 3c83f4847492417a91fb69cc7afc616a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.070583 17046 tablet_replica.cc:330] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.071040 17046 raft_consensus.cc:2229] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.071435 17046 raft_consensus.cc:2258] T 8e8576363f2f44f7b349932695b7fa58 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.073282 17046 tablet_replica.cc:330] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.073714 17046 raft_consensus.cc:2229] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.074154 17046 raft_consensus.cc:2258] T 98e78e9fa9e74827839975255f3f4d3b P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.075731 17046 tablet_replica.cc:330] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a: stopping tablet replica
I20240502 01:49:16.076177 17046 raft_consensus.cc:2229] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 4 LEADER]: Raft consensus shutting down.
I20240502 01:49:16.076980 17046 raft_consensus.cc:2258] T d761bc3026844a54822528ce6e783df8 P 3c83f4847492417a91fb69cc7afc616a [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.104826 17046 tablet_server.cc:194] TabletServer@127.16.165.131:0 shutdown complete.
I20240502 01:49:16.121407 17046 master.cc:537] Master@127.16.165.190:42449 shutting down...
I20240502 01:49:16.303028 17046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:16.303640 17046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.304011 17046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 211216f22b6a415f83466576fe691493: stopping tablet replica
I20240502 01:49:16.321544 17046 master.cc:552] Master@127.16.165.190:42449 shutdown complete.
3c83f4847492417a91fb69cc7afc616a 1
6fa7b2d1ea57421ba758a4231c9bcaf0 4
b759976c05b94fdc8be9d54a67b2c1b3 4
3c83f4847492417a91fb69cc7afc616a 3
6fa7b2d1ea57421ba758a4231c9bcaf0 3
b759976c05b94fdc8be9d54a67b2c1b3 3
3c83f4847492417a91fb69cc7afc616a 1
6fa7b2d1ea57421ba758a4231c9bcaf0 0
b759976c05b94fdc8be9d54a67b2c1b3 8
3c83f4847492417a91fb69cc7afc616a 3
6fa7b2d1ea57421ba758a4231c9bcaf0 3
b759976c05b94fdc8be9d54a67b2c1b3 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30918 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240502 01:49:16.361109 17046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.165.190:35283
I20240502 01:49:16.362247 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:16.368345 17571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.368930 17572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.369473 17574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:16.370071 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:16.371008 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:16.371208 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:16.371356 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614556371332 us; error 0 us; skew 500 ppm
I20240502 01:49:16.371850 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:16.374295 17046 webserver.cc:460] Webserver started at http://127.16.165.190:35493/ using document root <none> and password file <none>
I20240502 01:49:16.374900 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:16.375095 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:16.375437 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:16.377012 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/master-0-root/instance:
uuid: "dd1d760441cb46eda6176b35bb949d58"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.381896 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:16.385454 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.386236 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:49:16.386534 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/master-0-root
uuid: "dd1d760441cb46eda6176b35bb949d58"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.386812 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:16.407572 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:16.408929 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:16.446020 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.190:35283
I20240502 01:49:16.446132 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.190:35283 every 8 connection(s)
I20240502 01:49:16.449973 17632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.459993 17632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58: Bootstrap starting.
I20240502 01:49:16.464422 17632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:16.468742 17632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58: No bootstrap required, opened a new log
I20240502 01:49:16.470803 17632 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d760441cb46eda6176b35bb949d58" member_type: VOTER }
I20240502 01:49:16.471297 17632 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:16.471534 17632 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1d760441cb46eda6176b35bb949d58, State: Initialized, Role: FOLLOWER
I20240502 01:49:16.472090 17632 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d760441cb46eda6176b35bb949d58" member_type: VOTER }
I20240502 01:49:16.472551 17632 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:49:16.472770 17632 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:49:16.473042 17632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:16.477381 17632 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d760441cb46eda6176b35bb949d58" member_type: VOTER }
I20240502 01:49:16.477954 17632 leader_election.cc:302] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd1d760441cb46eda6176b35bb949d58; no voters:
I20240502 01:49:16.479122 17632 leader_election.cc:288] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:49:16.479570 17635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:16.480937 17635 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 1 LEADER]: Becoming Leader. State: Replica: dd1d760441cb46eda6176b35bb949d58, State: Running, Role: LEADER
I20240502 01:49:16.481675 17635 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d760441cb46eda6176b35bb949d58" member_type: VOTER }
I20240502 01:49:16.482030 17632 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:49:16.487114 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd1d760441cb46eda6176b35bb949d58. Latest consensus state: current_term: 1 leader_uuid: "dd1d760441cb46eda6176b35bb949d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d760441cb46eda6176b35bb949d58" member_type: VOTER } }
I20240502 01:49:16.487200 17636 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd1d760441cb46eda6176b35bb949d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d760441cb46eda6176b35bb949d58" member_type: VOTER } }
I20240502 01:49:16.488257 17636 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:16.488265 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:16.492619 17643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:49:16.500473 17643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:49:16.501550 17046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:49:16.509415 17643 catalog_manager.cc:1347] Generated new cluster ID: 091e83ff77ff4910a9b628432e370256
I20240502 01:49:16.509752 17643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:49:16.523706 17643 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:49:16.525029 17643 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:49:16.548457 17643 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58: Generated new TSK 0
I20240502 01:49:16.549105 17643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:49:16.568091 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:16.574532 17653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.575448 17654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.577288 17656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:16.578248 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:16.579464 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:16.579717 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:16.579931 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614556579912 us; error 0 us; skew 500 ppm
I20240502 01:49:16.580513 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:16.583984 17046 webserver.cc:460] Webserver started at http://127.16.165.129:46397/ using document root <none> and password file <none>
I20240502 01:49:16.584671 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:16.584919 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:16.585188 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:16.586688 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-0-root/instance:
uuid: "25eb7db9986849549a52ca275d122dbb"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.593160 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 01:49:16.597815 17661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.598901 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:49:16.599313 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-0-root
uuid: "25eb7db9986849549a52ca275d122dbb"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.599725 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:16.615047 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:16.616662 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:16.618636 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:16.621796 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:16.622073 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.622359 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:16.622568 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.661610 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.129:37049
I20240502 01:49:16.661705 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.129:37049 every 8 connection(s)
I20240502 01:49:16.666153 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:16.674492 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.675603 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:16.679138 17046 server_base.cc:1034] running on GCE node
W20240502 01:49:16.679428 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:16.680544 17727 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35283
I20240502 01:49:16.680959 17727 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:16.680928 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:16.681375 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:16.681577 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614556681559 us; error 0 us; skew 500 ppm
I20240502 01:49:16.682049 17727 heartbeater.cc:511] Master 127.16.165.190:35283 requested a full tablet report, sending...
I20240502 01:49:16.682394 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:16.684664 17597 ts_manager.cc:194] Registered new tserver with Master: 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049)
I20240502 01:49:16.685540 17046 webserver.cc:460] Webserver started at http://127.16.165.130:38773/ using document root <none> and password file <none>
I20240502 01:49:16.686208 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:16.686475 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:16.686791 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:16.687201 17597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41780
I20240502 01:49:16.688491 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-1-root/instance:
uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.693645 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:16.697149 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.697966 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:49:16.698254 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-1-root
uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.698556 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:16.716820 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:16.718079 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:16.719506 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:16.721701 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:16.721909 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.722126 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:16.722272 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.760311 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.130:36169
I20240502 01:49:16.760401 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.130:36169 every 8 connection(s)
I20240502 01:49:16.764823 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:16.774842 17807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.775650 17806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:16.779708 17809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:16.780054 17803 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35283
I20240502 01:49:16.780473 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:16.780512 17803 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:16.781560 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:49:16.781656 17803 heartbeater.cc:511] Master 127.16.165.190:35283 requested a full tablet report, sending...
W20240502 01:49:16.781868 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:16.782094 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614556782076 us; error 0 us; skew 500 ppm
I20240502 01:49:16.782828 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:16.783901 17597 ts_manager.cc:194] Registered new tserver with Master: e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:16.785467 17597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41786
I20240502 01:49:16.785542 17046 webserver.cc:460] Webserver started at http://127.16.165.131:46379/ using document root <none> and password file <none>
I20240502 01:49:16.786060 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:16.786259 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:16.786518 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:16.788097 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-2-root/instance:
uuid: "8c03ff01971e4877865c9bfa32d01da8"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.792434 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:49:16.795624 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.796437 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:49:16.796718 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-2-root
uuid: "8c03ff01971e4877865c9bfa32d01da8"
format_stamp: "Formatted at 2024-05-02 01:49:16 on dist-test-slave-kj4b"
I20240502 01:49:16.796993 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714614525285352-17046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:16.815996 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:16.817198 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:16.818683 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:16.820991 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:16.821190 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.821440 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:16.821585 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:16.860720 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.131:34205
I20240502 01:49:16.860847 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.131:34205 every 8 connection(s)
I20240502 01:49:16.874155 17879 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35283
I20240502 01:49:16.874563 17879 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:16.875372 17879 heartbeater.cc:511] Master 127.16.165.190:35283 requested a full tablet report, sending...
I20240502 01:49:16.877302 17597 ts_manager.cc:194] Registered new tserver with Master: 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:16.877470 17046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013475195s
I20240502 01:49:16.879072 17597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41802
I20240502 01:49:16.880489 17046 test_util.cc:268] Using random seed: -812192178
I20240502 01:49:16.903647 17597 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41810:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:49:16.906558 17597 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:49:16.974974 17768 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee3808f70484d27b5d18bcbb791e565 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 01:49:16.976678 17768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee3808f70484d27b5d18bcbb791e565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.982401 17767 tablet_service.cc:1467] Processing CreateTablet for tablet 425497d573154f9195ad16f6d2ee5611 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 01:49:16.983050 17765 tablet_service.cc:1467] Processing CreateTablet for tablet b76843eda8e4429cbe422ae6cb3f0b45 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 01:49:16.983992 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425497d573154f9195ad16f6d2ee5611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.982401 17766 tablet_service.cc:1467] Processing CreateTablet for tablet 8529aed8c4c14bdd870a4d23d857baf2 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 01:49:16.984210 17689 tablet_service.cc:1467] Processing CreateTablet for tablet 8529aed8c4c14bdd870a4d23d857baf2 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 01:49:16.984829 17690 tablet_service.cc:1467] Processing CreateTablet for tablet 425497d573154f9195ad16f6d2ee5611 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 01:49:16.985311 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76843eda8e4429cbe422ae6cb3f0b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.985790 17689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529aed8c4c14bdd870a4d23d857baf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.987157 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425497d573154f9195ad16f6d2ee5611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.988886 17688 tablet_service.cc:1467] Processing CreateTablet for tablet b76843eda8e4429cbe422ae6cb3f0b45 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 01:49:16.990258 17766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529aed8c4c14bdd870a4d23d857baf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.990725 17688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76843eda8e4429cbe422ae6cb3f0b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.983517 17691 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee3808f70484d27b5d18bcbb791e565 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 01:49:16.992687 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee3808f70484d27b5d18bcbb791e565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.995688 17764 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8e64a268154607acf2007372270c14 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 01:49:16.997113 17764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8e64a268154607acf2007372270c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:16.997594 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8e64a268154607acf2007372270c14 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 01:49:16.998873 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8e64a268154607acf2007372270c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.004396 17763 tablet_service.cc:1467] Processing CreateTablet for tablet db9db6c971444af2a9fdc56330f4cf4f (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 01:49:17.005774 17763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9db6c971444af2a9fdc56330f4cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.023142 17762 tablet_service.cc:1467] Processing CreateTablet for tablet 2196a1b679d94d769f5593718e77173b (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 01:49:17.024607 17762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2196a1b679d94d769f5593718e77173b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.025947 17686 tablet_service.cc:1467] Processing CreateTablet for tablet db9db6c971444af2a9fdc56330f4cf4f (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 01:49:17.027287 17686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9db6c971444af2a9fdc56330f4cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.029119 17685 tablet_service.cc:1467] Processing CreateTablet for tablet 2196a1b679d94d769f5593718e77173b (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 01:49:17.030411 17685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2196a1b679d94d769f5593718e77173b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.042634 17687 tablet_service.cc:1467] Processing CreateTablet for tablet e314bdfd1de844dca1ca9bacc6ac5778 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 01:49:17.043987 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e314bdfd1de844dca1ca9bacc6ac5778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.054024 17761 tablet_service.cc:1467] Processing CreateTablet for tablet e314bdfd1de844dca1ca9bacc6ac5778 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 01:49:17.055435 17761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e314bdfd1de844dca1ca9bacc6ac5778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.057112 17684 tablet_service.cc:1467] Processing CreateTablet for tablet e8803ce7fc814fc1b7e9058cb542ca3c (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 01:49:17.058455 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8803ce7fc814fc1b7e9058cb542ca3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.060717 17843 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee3808f70484d27b5d18bcbb791e565 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 01:49:17.062028 17843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee3808f70484d27b5d18bcbb791e565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.065644 17842 tablet_service.cc:1467] Processing CreateTablet for tablet 425497d573154f9195ad16f6d2ee5611 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 01:49:17.067009 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425497d573154f9195ad16f6d2ee5611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.069658 17841 tablet_service.cc:1467] Processing CreateTablet for tablet 8529aed8c4c14bdd870a4d23d857baf2 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 01:49:17.071019 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529aed8c4c14bdd870a4d23d857baf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.072940 17763 tablet_service.cc:1467] Processing CreateTablet for tablet e8803ce7fc814fc1b7e9058cb542ca3c (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 01:49:17.073060 17840 tablet_service.cc:1467] Processing CreateTablet for tablet b76843eda8e4429cbe422ae6cb3f0b45 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 01:49:17.074265 17763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8803ce7fc814fc1b7e9058cb542ca3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.074435 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76843eda8e4429cbe422ae6cb3f0b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.076977 17894 tablet_bootstrap.cc:492] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:17.077086 17839 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8e64a268154607acf2007372270c14 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 01:49:17.078495 17839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8e64a268154607acf2007372270c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.081199 17838 tablet_service.cc:1467] Processing CreateTablet for tablet db9db6c971444af2a9fdc56330f4cf4f (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 01:49:17.082578 17838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9db6c971444af2a9fdc56330f4cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.083093 17894 tablet_bootstrap.cc:654] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.084865 17691 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9467a8fb444f8baaf3349f68bec2cb (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 01:49:17.086215 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9467a8fb444f8baaf3349f68bec2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.086673 17836 tablet_service.cc:1467] Processing CreateTablet for tablet e314bdfd1de844dca1ca9bacc6ac5778 (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 01:49:17.089093 17835 tablet_service.cc:1467] Processing CreateTablet for tablet e8803ce7fc814fc1b7e9058cb542ca3c (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 01:49:17.084980 17837 tablet_service.cc:1467] Processing CreateTablet for tablet 2196a1b679d94d769f5593718e77173b (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 01:49:17.091087 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e314bdfd1de844dca1ca9bacc6ac5778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.092814 17766 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9467a8fb444f8baaf3349f68bec2cb (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 01:49:17.094096 17766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9467a8fb444f8baaf3349f68bec2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.097211 17834 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9467a8fb444f8baaf3349f68bec2cb (DEFAULT_TABLE table=test-workload [id=54422a22e79844e5a78fd34762f5dd46]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 01:49:17.098587 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9467a8fb444f8baaf3349f68bec2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.100215 17837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2196a1b679d94d769f5593718e77173b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.102308 17835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8803ce7fc814fc1b7e9058cb542ca3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:17.160589 17897 tablet_bootstrap.cc:492] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:17.171623 17897 tablet_bootstrap.cc:654] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.190927 17896 tablet_bootstrap.cc:492] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:17.192947 17894 tablet_bootstrap.cc:492] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:17.193521 17894 ts_tablet_manager.cc:1367] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.117s user 0.042s sys 0.028s
I20240502 01:49:17.193984 17897 tablet_bootstrap.cc:492] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:17.194455 17897 ts_tablet_manager.cc:1367] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.004s
I20240502 01:49:17.197405 17897 raft_consensus.cc:348] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.198201 17894 raft_consensus.cc:348] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.198963 17894 raft_consensus.cc:374] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.198967 17897 raft_consensus.cc:374] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.199424 17894 raft_consensus.cc:729] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.199470 17897 raft_consensus.cc:729] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.200857 17896 tablet_bootstrap.cc:654] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.200345 17897 consensus_queue.cc:260] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.200237 17894 consensus_queue.cc:260] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.202901 17727 heartbeater.cc:503] Master 127.16.165.190:35283 was elected leader, sending a full tablet report...
I20240502 01:49:17.203541 17897 ts_tablet_manager.cc:1398] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:49:17.204164 17803 heartbeater.cc:503] Master 127.16.165.190:35283 was elected leader, sending a full tablet report...
I20240502 01:49:17.204561 17897 tablet_bootstrap.cc:492] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:17.206645 17896 tablet_bootstrap.cc:492] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:17.207139 17896 ts_tablet_manager.cc:1367] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.003s
I20240502 01:49:17.208789 17894 ts_tablet_manager.cc:1398] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.015s user 0.002s sys 0.004s
I20240502 01:49:17.209889 17894 tablet_bootstrap.cc:492] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:17.210594 17897 tablet_bootstrap.cc:654] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.210085 17896 raft_consensus.cc:348] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.211076 17896 raft_consensus.cc:374] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.211438 17896 raft_consensus.cc:729] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.212430 17896 consensus_queue.cc:260] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.217124 17894 tablet_bootstrap.cc:654] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.221783 17879 heartbeater.cc:503] Master 127.16.165.190:35283 was elected leader, sending a full tablet report...
I20240502 01:49:17.222388 17896 ts_tablet_manager.cc:1398] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.015s user 0.005s sys 0.012s
I20240502 01:49:17.223475 17896 tablet_bootstrap.cc:492] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:17.223737 17897 tablet_bootstrap.cc:492] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:17.224310 17897 ts_tablet_manager.cc:1367] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20240502 01:49:17.227200 17897 raft_consensus.cc:348] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.228083 17897 raft_consensus.cc:374] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.228415 17897 raft_consensus.cc:729] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.229661 17896 tablet_bootstrap.cc:654] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.229136 17897 consensus_queue.cc:260] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.232187 17897 ts_tablet_manager.cc:1398] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240502 01:49:17.233435 17897 tablet_bootstrap.cc:492] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:17.241425 17897 tablet_bootstrap.cc:654] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.260601 17897 tablet_bootstrap.cc:492] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:17.260601 17896 tablet_bootstrap.cc:492] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:17.261265 17897 ts_tablet_manager.cc:1367] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.011s
I20240502 01:49:17.261416 17896 ts_tablet_manager.cc:1367] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.005s
I20240502 01:49:17.263571 17896 raft_consensus.cc:348] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.264254 17896 raft_consensus.cc:374] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.264173 17897 raft_consensus.cc:348] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.264667 17896 raft_consensus.cc:729] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.265075 17897 raft_consensus.cc:374] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.265417 17897 raft_consensus.cc:729] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.265692 17896 consensus_queue.cc:260] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.266113 17897 consensus_queue.cc:260] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.267529 17896 ts_tablet_manager.cc:1398] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240502 01:49:17.268380 17896 tablet_bootstrap.cc:492] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:17.268525 17897 ts_tablet_manager.cc:1398] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20240502 01:49:17.269558 17897 tablet_bootstrap.cc:492] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:17.272603 17896 tablet_bootstrap.cc:654] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.275491 17897 tablet_bootstrap.cc:654] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.278219 17896 tablet_bootstrap.cc:492] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:17.279490 17896 ts_tablet_manager.cc:1367] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240502 01:49:17.281424 17897 tablet_bootstrap.cc:492] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:17.281805 17897 ts_tablet_manager.cc:1367] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20240502 01:49:17.283114 17896 raft_consensus.cc:348] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.283977 17896 raft_consensus.cc:374] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.283931 17897 raft_consensus.cc:348] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.284440 17896 raft_consensus.cc:729] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.284658 17897 raft_consensus.cc:374] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:17.284981 17897 raft_consensus.cc:729] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:17.285291 17896 consensus_queue.cc:260] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.285605 17897 consensus_queue.cc:260] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.287770 17896 ts_tablet_manager.cc:1398] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240502 01:49:17.288498 17897 ts_tablet_manager.cc:1398] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:49:17.289038 17896 tablet_bootstrap.cc:492] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:17.289304 17897 tablet_bootstrap.cc:492] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:17.295498 17896 tablet_bootstrap.cc:654] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.295498 17897 tablet_bootstrap.cc:654] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:17.979705 17899 raft_consensus.cc:482] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:17.980330 17899 raft_consensus.cc:504] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.982280 17899 leader_election.cc:288] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:17.994967 17912 raft_consensus.cc:482] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:17.995566 17912 raft_consensus.cc:504] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:17.998050 17912 leader_election.cc:288] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:17.999217 17899 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:17.999740 17899 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.002278 17899 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.013253 17894 tablet_bootstrap.cc:492] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.013864 17894 ts_tablet_manager.cc:1367] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.804s user 0.020s sys 0.026s
W20240502 01:49:18.016681 17897 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.718s user 0.005s sys 0.002s
I20240502 01:49:18.016716 17894 raft_consensus.cc:348] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.017709 17894 raft_consensus.cc:374] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.018007 17894 raft_consensus.cc:729] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.018800 17897 tablet_bootstrap.cc:492] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:18.019196 17897 ts_tablet_manager.cc:1367] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.730s user 0.011s sys 0.007s
I20240502 01:49:18.018697 17894 consensus_queue.cc:260] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.020895 17894 ts_tablet_manager.cc:1398] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:49:18.021344 17897 raft_consensus.cc:348] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.021770 17894 tablet_bootstrap.cc:492] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.021878 17897 raft_consensus.cc:374] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.022207 17897 raft_consensus.cc:729] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.023224 17916 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.022836 17897 consensus_queue.cc:260] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.023722 17916 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.025694 17897 ts_tablet_manager.cc:1398] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 01:49:18.026355 17916 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.026646 17897 tablet_bootstrap.cc:492] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:18.026954 17916 raft_consensus.cc:482] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.027439 17916 raft_consensus.cc:504] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.029304 17916 leader_election.cc:288] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.029827 17916 raft_consensus.cc:482] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 01:49:18.026968 17896 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.003s sys 0.014s
W20240502 01:49:18.030289 17896 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.732s user 0.004s sys 0.015s
I20240502 01:49:18.030776 17916 raft_consensus.cc:504] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.032531 17896 tablet_bootstrap.cc:492] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.032595 17916 leader_election.cc:288] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049)
I20240502 01:49:18.033063 17896 ts_tablet_manager.cc:1367] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.744s user 0.012s sys 0.015s
I20240502 01:49:18.033747 17894 tablet_bootstrap.cc:654] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.035035 17896 raft_consensus.cc:348] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.035594 17896 raft_consensus.cc:374] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.035857 17896 raft_consensus.cc:729] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.036681 17896 consensus_queue.cc:260] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.038960 17896 ts_tablet_manager.cc:1398] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:49:18.044787 17896 tablet_bootstrap.cc:492] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:18.048381 17897 tablet_bootstrap.cc:654] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.049747 17894 tablet_bootstrap.cc:492] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.050308 17894 ts_tablet_manager.cc:1367] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.001s
I20240502 01:49:18.069346 17896 tablet_bootstrap.cc:654] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:18.072372 17645 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:18.077245 17897 tablet_bootstrap.cc:492] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:18.077816 17897 ts_tablet_manager.cc:1367] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.007s
I20240502 01:49:18.081512 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.083614 17777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.084005 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.084442 17777 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.084739 17776 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.081640 17894 raft_consensus.cc:348] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.085963 17817 leader_election.cc:302] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.086061 17894 raft_consensus.cc:374] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.086979 17894 raft_consensus.cc:729] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.087158 17903 raft_consensus.cc:2781] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.087611 17903 raft_consensus.cc:482] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.088034 17903 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.090056 17894 consensus_queue.cc:260] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.091153 17778 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.092738 17817 leader_election.cc:302] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.093787 17916 raft_consensus.cc:2781] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.094214 17916 raft_consensus.cc:482] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.094564 17916 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.104439 17817 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.108772 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.109649 17702 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 0.
I20240502 01:49:18.111238 17741 leader_election.cc:302] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.112529 17912 raft_consensus.cc:2781] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.112933 17912 raft_consensus.cc:482] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.113281 17912 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.119277 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.089447 17897 raft_consensus.cc:348] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.122012 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.122310 17897 raft_consensus.cc:374] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.122690 17897 raft_consensus.cc:729] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.122756 17776 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.124234 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.125584 17894 ts_tablet_manager.cc:1398] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.075s user 0.005s sys 0.005s
I20240502 01:49:18.125460 17912 raft_consensus.cc:504] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.134878 17896 tablet_bootstrap.cc:492] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.135715 17896 ts_tablet_manager.cc:1367] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.006s
I20240502 01:49:18.135663 17894 tablet_bootstrap.cc:492] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.138615 17896 raft_consensus.cc:348] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.127172 17903 raft_consensus.cc:504] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.140214 17853 raft_consensus.cc:2379] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in current term 1: Already voted for candidate 8c03ff01971e4877865c9bfa32d01da8 in this term.
I20240502 01:49:18.140631 17912 leader_election.cc:288] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.141829 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.142179 17894 tablet_bootstrap.cc:654] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.129005 17927 raft_consensus.cc:2781] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.144109 17927 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.144516 17927 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.145310 17896 raft_consensus.cc:374] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.145627 17896 raft_consensus.cc:729] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.146042 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.146914 17778 raft_consensus.cc:2379] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c03ff01971e4877865c9bfa32d01da8 in current term 1: Already voted for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in this term.
I20240502 01:49:18.128708 17897 consensus_queue.cc:260] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.131981 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.132278 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.138918 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.148772 17702 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.150092 17894 tablet_bootstrap.cc:492] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.150578 17894 ts_tablet_manager.cc:1367] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
W20240502 01:49:18.152248 17663 leader_election.cc:341] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:18.126448 17916 raft_consensus.cc:504] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.152946 17894 raft_consensus.cc:348] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.153504 17894 raft_consensus.cc:374] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.154002 17916 leader_election.cc:288] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049)
I20240502 01:49:18.154902 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.155563 17778 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.153740 17894 raft_consensus.cc:729] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.156955 17702 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 1.
I20240502 01:49:18.157951 17897 ts_tablet_manager.cc:1398] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.080s user 0.005s sys 0.005s
I20240502 01:49:18.157572 17894 consensus_queue.cc:260] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.158696 17897 tablet_bootstrap.cc:492] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:18.159399 17741 leader_election.cc:302] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters: 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:18.159996 17903 leader_election.cc:288] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.130385 17664 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
W20240502 01:49:18.161049 17664 leader_election.cc:341] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:18.161497 17664 leader_election.cc:302] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb; no voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:18.162575 17899 raft_consensus.cc:2781] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.146342 17896 consensus_queue.cc:260] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.163312 17912 raft_consensus.cc:2781] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.163780 17912 raft_consensus.cc:686] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:18.164793 17897 tablet_bootstrap.cc:654] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.164937 17896 ts_tablet_manager.cc:1398] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20240502 01:49:18.164630 17912 consensus_queue.cc:237] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.165921 17896 tablet_bootstrap.cc:492] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:18.162918 17899 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.167708 17899 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.171747 17894 ts_tablet_manager.cc:1398] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20240502 01:49:18.172873 17896 tablet_bootstrap.cc:654] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.172940 17894 tablet_bootstrap.cc:492] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.183220 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.183943 17894 tablet_bootstrap.cc:654] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.185146 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.185487 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.186084 17930 raft_consensus.cc:2726] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:18.186158 17701 raft_consensus.cc:2379] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c03ff01971e4877865c9bfa32d01da8 in current term 1: Already voted for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in this term.
I20240502 01:49:18.186326 17700 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.187953 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.188848 17700 raft_consensus.cc:2379] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c03ff01971e4877865c9bfa32d01da8 in current term 1: Already voted for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in this term.
I20240502 01:49:18.190328 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.190994 17700 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.193382 17816 leader_election.cc:302] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8; no voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:18.194780 17903 raft_consensus.cc:2726] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:18.192390 17899 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.204113 17702 raft_consensus.cc:2379] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c03ff01971e4877865c9bfa32d01da8 in current term 1: Already voted for candidate 25eb7db9986849549a52ca275d122dbb in this term.
I20240502 01:49:18.200332 17778 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.206389 17817 leader_election.cc:302] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.190714 17927 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.207301 17903 raft_consensus.cc:2781] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.208053 17903 raft_consensus.cc:686] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:18.209239 17851 raft_consensus.cc:2379] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 25eb7db9986849549a52ca275d122dbb in current term 1: Already voted for candidate 8c03ff01971e4877865c9bfa32d01da8 in this term.
I20240502 01:49:18.209821 17903 consensus_queue.cc:237] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.213320 17899 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.214221 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.217741 17896 tablet_bootstrap.cc:492] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.222663 17595 catalog_manager.cc:5484] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 1 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:18.213394 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.218537 17700 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.222849 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.214538 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.224021 17896 ts_tablet_manager.cc:1367] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.058s user 0.018s sys 0.002s
I20240502 01:49:18.223994 17702 raft_consensus.cc:2379] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c03ff01971e4877865c9bfa32d01da8 in current term 1: Already voted for candidate 25eb7db9986849549a52ca275d122dbb in this term.
I20240502 01:49:18.224725 17851 raft_consensus.cc:2379] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 25eb7db9986849549a52ca275d122dbb in current term 1: Already voted for candidate 8c03ff01971e4877865c9bfa32d01da8 in this term.
I20240502 01:49:18.224718 17776 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.223400 17927 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.227591 17896 raft_consensus.cc:348] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.228391 17896 raft_consensus.cc:374] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.228699 17896 raft_consensus.cc:729] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.229401 17896 consensus_queue.cc:260] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.223301 17897 tablet_bootstrap.cc:492] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:18.231518 17897 ts_tablet_manager.cc:1367] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.073s user 0.029s sys 0.029s
I20240502 01:49:18.232717 17776 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.233906 17896 ts_tablet_manager.cc:1398] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:49:18.233681 17817 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters: 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:18.234516 17927 raft_consensus.cc:2781] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.234705 17896 tablet_bootstrap.cc:492] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:18.235040 17927 raft_consensus.cc:686] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:18.235205 17778 raft_consensus.cc:2379] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 25eb7db9986849549a52ca275d122dbb in current term 1: Already voted for candidate 8c03ff01971e4877865c9bfa32d01da8 in this term.
I20240502 01:49:18.236133 17927 consensus_queue.cc:237] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.237998 17894 tablet_bootstrap.cc:492] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.238476 17894 ts_tablet_manager.cc:1367] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.022s
I20240502 01:49:18.238310 17664 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb; no voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:18.239404 17899 raft_consensus.cc:2726] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:18.240856 17896 tablet_bootstrap.cc:654] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.241190 17894 raft_consensus.cc:348] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.241969 17894 raft_consensus.cc:374] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.242255 17894 raft_consensus.cc:729] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.242966 17894 consensus_queue.cc:260] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.234200 17897 raft_consensus.cc:348] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.244853 17897 raft_consensus.cc:374] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.245965 17897 raft_consensus.cc:729] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.245968 17894 ts_tablet_manager.cc:1398] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:49:18.247758 17894 tablet_bootstrap.cc:492] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.247519 17897 consensus_queue.cc:260] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.252346 17897 ts_tablet_manager.cc:1398] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20240502 01:49:18.253417 17894 tablet_bootstrap.cc:654] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.253509 17897 tablet_bootstrap.cc:492] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:18.253407 17896 tablet_bootstrap.cc:492] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.256057 17896 ts_tablet_manager.cc:1367] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.000s
I20240502 01:49:18.258975 17896 raft_consensus.cc:348] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.260603 17896 raft_consensus.cc:374] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.259979 17897 tablet_bootstrap.cc:654] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.261026 17896 raft_consensus.cc:729] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.261920 17896 consensus_queue.cc:260] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.264350 17896 ts_tablet_manager.cc:1398] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:49:18.265481 17896 tablet_bootstrap.cc:492] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:18.265156 17594 catalog_manager.cc:5484] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 1 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:18.272472 17896 tablet_bootstrap.cc:654] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.282267 17927 raft_consensus.cc:482] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.282917 17927 raft_consensus.cc:504] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.285120 17927 leader_election.cc:288] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.286839 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.287607 17700 raft_consensus.cc:2454] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.289119 17816 leader_election.cc:302] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:18.290220 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.290457 17927 raft_consensus.cc:2781] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.290815 17927 raft_consensus.cc:482] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.291137 17927 raft_consensus.cc:3037] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:18.291771 17817 leader_election.cc:341] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:18.292217 17896 tablet_bootstrap.cc:492] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.292716 17896 ts_tablet_manager.cc:1367] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.018s
I20240502 01:49:18.294180 17897 tablet_bootstrap.cc:492] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:18.294656 17897 ts_tablet_manager.cc:1367] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.014s
I20240502 01:49:18.295291 17896 raft_consensus.cc:348] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.296015 17896 raft_consensus.cc:374] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.296267 17896 raft_consensus.cc:729] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.296638 17927 raft_consensus.cc:504] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.296885 17896 consensus_queue.cc:260] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.297240 17897 raft_consensus.cc:348] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.297870 17897 raft_consensus.cc:374] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.298136 17897 raft_consensus.cc:729] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.298966 17896 ts_tablet_manager.cc:1398] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:49:18.299688 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.299559 17897 consensus_queue.cc:260] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.300375 17700 raft_consensus.cc:3037] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.300931 17894 tablet_bootstrap.cc:492] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.301417 17894 ts_tablet_manager.cc:1367] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.025s
I20240502 01:49:18.301927 17896 tablet_bootstrap.cc:492] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:18.304126 17894 raft_consensus.cc:348] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.304966 17894 raft_consensus.cc:374] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.305289 17894 raft_consensus.cc:729] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.306097 17894 consensus_queue.cc:260] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.306291 17594 catalog_manager.cc:5484] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 1 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:18.307767 17896 tablet_bootstrap.cc:654] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.308610 17894 ts_tablet_manager.cc:1398] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20240502 01:49:18.309664 17723 debug-util.cc:398] Leaking SignalData structure 0x7b08001ab740 after lost signal to thread 17049
I20240502 01:49:18.309967 17894 tablet_bootstrap.cc:492] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
W20240502 01:49:18.310571 17723 debug-util.cc:398] Leaking SignalData structure 0x7b08001a4ee0 after lost signal to thread 17631
I20240502 01:49:18.318490 17894 tablet_bootstrap.cc:654] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.319263 17700 raft_consensus.cc:2454] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.320642 17897 ts_tablet_manager.cc:1398] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20240502 01:49:18.320513 17816 leader_election.cc:302] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:18.321651 17903 raft_consensus.cc:2781] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.322141 17903 raft_consensus.cc:686] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:18.323534 17897 tablet_bootstrap.cc:492] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
I20240502 01:49:18.323009 17903 consensus_queue.cc:237] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.330799 17894 tablet_bootstrap.cc:492] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.331377 17894 ts_tablet_manager.cc:1367] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.005s
I20240502 01:49:18.334059 17894 raft_consensus.cc:348] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.334961 17894 raft_consensus.cc:374] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.335259 17894 raft_consensus.cc:729] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.335175 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.335341 17927 leader_election.cc:288] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.336026 17778 raft_consensus.cc:3037] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.336273 17894 consensus_queue.cc:260] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.337090 17897 tablet_bootstrap.cc:654] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.338711 17894 ts_tablet_manager.cc:1398] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:49:18.339790 17894 tablet_bootstrap.cc:492] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.343716 17778 raft_consensus.cc:2454] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.347425 17894 tablet_bootstrap.cc:654] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.349962 17896 tablet_bootstrap.cc:492] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.350490 17896 ts_tablet_manager.cc:1367] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.020s
I20240502 01:49:18.351562 17912 raft_consensus.cc:482] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.352136 17912 raft_consensus.cc:504] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.353619 17896 raft_consensus.cc:348] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.354351 17896 raft_consensus.cc:374] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.354645 17896 raft_consensus.cc:729] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.355403 17896 consensus_queue.cc:260] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.355792 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.356508 17851 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 0.
I20240502 01:49:18.357609 17896 ts_tablet_manager.cc:1398] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:49:18.357995 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.358570 17896 tablet_bootstrap.cc:492] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8: Bootstrap starting.
I20240502 01:49:18.358681 17700 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 0.
I20240502 01:49:18.360352 17741 leader_election.cc:302] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.361617 17928 raft_consensus.cc:2781] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.362133 17928 raft_consensus.cc:482] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.362648 17928 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.364427 17896 tablet_bootstrap.cc:654] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.354142 17912 leader_election.cc:288] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.370815 17928 raft_consensus.cc:504] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.370689 17593 catalog_manager.cc:5484] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 1 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:18.373853 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.374500 17700 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.378419 17897 tablet_bootstrap.cc:492] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:18.378964 17897 ts_tablet_manager.cc:1367] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.056s user 0.015s sys 0.002s
I20240502 01:49:18.381584 17897 raft_consensus.cc:348] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.382297 17897 raft_consensus.cc:374] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.383853 17897 raft_consensus.cc:729] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.384662 17897 consensus_queue.cc:260] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.387379 17700 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 1.
I20240502 01:49:18.388538 17928 leader_election.cc:288] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.389607 17741 leader_election.cc:302] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.390455 17928 raft_consensus.cc:2781] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.391204 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.391638 17928 raft_consensus.cc:686] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:18.391808 17851 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.392527 17928 consensus_queue.cc:237] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.402482 17851 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 1.
I20240502 01:49:18.408179 17897 ts_tablet_manager.cc:1398] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20240502 01:49:18.409133 17595 catalog_manager.cc:5484] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 1 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:18.410285 17894 tablet_bootstrap.cc:492] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.411067 17894 ts_tablet_manager.cc:1367] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.006s
I20240502 01:49:18.413829 17894 raft_consensus.cc:348] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.414598 17894 raft_consensus.cc:374] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.414922 17894 raft_consensus.cc:729] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.415585 17894 consensus_queue.cc:260] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.416743 17897 tablet_bootstrap.cc:492] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: Bootstrap starting.
W20240502 01:49:18.417749 17880 tablet.cc:2369] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:49:18.419934 17894 ts_tablet_manager.cc:1398] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:49:18.420879 17894 tablet_bootstrap.cc:492] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.426714 17896 tablet_bootstrap.cc:492] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8: No bootstrap required, opened a new log
I20240502 01:49:18.427173 17896 ts_tablet_manager.cc:1367] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8: Time spent bootstrapping tablet: real 0.069s user 0.020s sys 0.044s
I20240502 01:49:18.429481 17897 tablet_bootstrap.cc:654] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.429689 17896 raft_consensus.cc:348] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.432196 17896 raft_consensus.cc:374] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.432511 17896 raft_consensus.cc:729] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.433282 17894 tablet_bootstrap.cc:654] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.433224 17896 consensus_queue.cc:260] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.438513 17896 ts_tablet_manager.cc:1398] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240502 01:49:18.445730 17897 tablet_bootstrap.cc:492] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: No bootstrap required, opened a new log
I20240502 01:49:18.446202 17897 ts_tablet_manager.cc:1367] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.005s
I20240502 01:49:18.448693 17897 raft_consensus.cc:348] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.449399 17897 raft_consensus.cc:374] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.449697 17897 raft_consensus.cc:729] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.450332 17897 consensus_queue.cc:260] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.451732 17894 tablet_bootstrap.cc:492] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.452207 17897 ts_tablet_manager.cc:1398] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:49:18.452248 17894 ts_tablet_manager.cc:1367] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.009s
I20240502 01:49:18.455610 17894 raft_consensus.cc:348] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.456355 17894 raft_consensus.cc:374] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.456691 17894 raft_consensus.cc:729] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.457353 17894 consensus_queue.cc:260] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.459686 17894 ts_tablet_manager.cc:1398] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:49:18.460649 17894 tablet_bootstrap.cc:492] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Bootstrap starting.
I20240502 01:49:18.466361 17894 tablet_bootstrap.cc:654] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:18.472029 17894 tablet_bootstrap.cc:492] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: No bootstrap required, opened a new log
I20240502 01:49:18.472502 17894 ts_tablet_manager.cc:1367] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 01:49:18.474882 17894 raft_consensus.cc:348] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.475602 17894 raft_consensus.cc:374] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:18.475891 17894 raft_consensus.cc:729] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Initialized, Role: FOLLOWER
I20240502 01:49:18.476672 17894 consensus_queue.cc:260] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.478807 17894 ts_tablet_manager.cc:1398] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:49:18.492602 17899 raft_consensus.cc:482] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.493093 17899 raft_consensus.cc:504] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.494778 17899 leader_election.cc:288] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.495733 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.495909 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.496313 17778 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.496601 17851 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.497678 17663 leader_election.cc:302] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:18.500361 17899 raft_consensus.cc:2781] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.500643 17899 raft_consensus.cc:482] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.500885 17899 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.505537 17899 raft_consensus.cc:504] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.506944 17899 leader_election.cc:288] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.507920 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.508044 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.508401 17778 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.508654 17851 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.512611 17778 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 1.
I20240502 01:49:18.513485 17664 leader_election.cc:302] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.514086 17899 raft_consensus.cc:2781] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.514443 17899 raft_consensus.cc:686] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:18.514772 17851 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 1.
I20240502 01:49:18.515189 17899 consensus_queue.cc:237] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.522552 17593 catalog_manager.cc:5484] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 1 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:18.532419 17899 raft_consensus.cc:482] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.532825 17899 raft_consensus.cc:504] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.534377 17899 leader_election.cc:288] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.535102 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.535440 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.535621 17778 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.536461 17851 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.536566 17664 leader_election.cc:302] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.537240 17899 raft_consensus.cc:2781] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.537612 17899 raft_consensus.cc:482] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.537925 17899 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.544306 17899 raft_consensus.cc:504] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.546141 17899 leader_election.cc:288] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.546823 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.547319 17778 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.547329 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.548048 17851 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.552083 17778 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 1.
I20240502 01:49:18.553095 17664 leader_election.cc:302] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.553841 17899 raft_consensus.cc:2781] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.554230 17899 raft_consensus.cc:686] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:18.554646 17851 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 1.
I20240502 01:49:18.555032 17899 consensus_queue.cc:237] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.562960 17593 catalog_manager.cc:5484] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 1 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:18.605434 17928 raft_consensus.cc:482] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.605904 17928 raft_consensus.cc:504] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.607496 17928 leader_election.cc:288] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.608429 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.608700 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.609001 17700 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 0.
I20240502 01:49:18.609445 17851 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 0.
I20240502 01:49:18.609962 17741 leader_election.cc:302] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.610725 17928 raft_consensus.cc:2781] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.611107 17928 raft_consensus.cc:482] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.611398 17928 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.616576 17928 raft_consensus.cc:504] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.617945 17903 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.618395 17928 leader_election.cc:288] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.619495 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.620086 17700 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.620031 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.620783 17851 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.626359 17928 consensus_queue.cc:1035] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.627681 17700 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 1.
I20240502 01:49:18.630720 17741 leader_election.cc:302] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.631582 17912 raft_consensus.cc:2781] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.632597 17912 raft_consensus.cc:686] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:18.633424 17912 consensus_queue.cc:237] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.638280 17851 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 1.
I20240502 01:49:18.658620 17899 raft_consensus.cc:482] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.659291 17899 raft_consensus.cc:504] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.662540 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.663304 17778 raft_consensus.cc:2454] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.664755 17664 leader_election.cc:302] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.666191 17930 raft_consensus.cc:2781] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.666582 17930 raft_consensus.cc:482] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.666919 17930 raft_consensus.cc:3037] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.674324 17912 consensus_queue.cc:1035] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.678625 17927 raft_consensus.cc:482] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:18.678748 17916 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.679136 17927 raft_consensus.cc:504] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.681102 17927 leader_election.cc:288] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.682303 17930 raft_consensus.cc:504] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.684280 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb" is_pre_election: true
I20240502 01:49:18.683132 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" is_pre_election: true
I20240502 01:49:18.685310 17776 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.685981 17899 leader_election.cc:288] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.687628 17930 leader_election.cc:288] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.686561 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8" is_pre_election: true
I20240502 01:49:18.688272 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.687947 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.688889 17776 raft_consensus.cc:3037] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.688905 17852 raft_consensus.cc:2454] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 0.
I20240502 01:49:18.686488 17817 leader_election.cc:302] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.693255 17903 raft_consensus.cc:2781] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:18.693717 17903 raft_consensus.cc:482] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:18.694082 17903 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.684938 17702 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 0.
I20240502 01:49:18.683856 17594 catalog_manager.cc:5484] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 1 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:18.707806 17927 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.715915 17903 raft_consensus.cc:504] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.717290 17776 raft_consensus.cc:2454] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 1.
I20240502 01:49:18.720597 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:18.721316 17702 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.724109 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.724815 17778 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:18.728547 17664 leader_election.cc:302] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters: 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:18.729552 17930 raft_consensus.cc:2781] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.730029 17930 raft_consensus.cc:686] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:18.730855 17930 consensus_queue.cc:237] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.737043 17927 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.749670 17778 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.741322 17903 leader_election.cc:288] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:18.755614 17817 leader_election.cc:302] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.758424 17702 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 1.
I20240502 01:49:18.759030 17903 raft_consensus.cc:2781] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:18.761003 17903 raft_consensus.cc:686] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:18.761900 17903 consensus_queue.cc:237] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.778592 17927 consensus_queue.cc:1035] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.794600 17595 catalog_manager.cc:5484] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 1 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:18.819816 17903 consensus_queue.cc:1035] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.820497 17594 catalog_manager.cc:5484] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 1 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:18.863816 17851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35374
I20240502 01:49:18.864611 17851 raft_consensus.cc:595] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:18.877619 17777 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:55816
I20240502 01:49:18.878137 17777 raft_consensus.cc:595] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:18.882905 17777 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:55816
I20240502 01:49:18.883414 17777 raft_consensus.cc:595] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:18.888120 17851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35374
I20240502 01:49:18.888790 17851 raft_consensus.cc:595] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:18.893905 17851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e8e64a268154607acf2007372270c14"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35374
I20240502 01:49:18.894556 17851 raft_consensus.cc:595] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:18.910352 17701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2196a1b679d94d769f5593718e77173b"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:35654
I20240502 01:49:18.911007 17701 raft_consensus.cc:595] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Received request to transfer leadership to TS e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:18.915917 17701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:35654
I20240502 01:49:18.916527 17701 raft_consensus.cc:595] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Received request to transfer leadership to TS e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:18.921149 17851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
mode: GRACEFUL
new_leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:35374
I20240502 01:49:18.921761 17851 raft_consensus.cc:595] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Received request to transfer leadership to TS e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:18.926393 17701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35654
I20240502 01:49:18.926986 17701 raft_consensus.cc:595] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Received request to transfer leadership to TS 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:18.948127 17912 consensus_queue.cc:1035] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.956163 17912 raft_consensus.cc:979] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:18.957398 17959 consensus_queue.cc:1035] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.959954 17928 raft_consensus.cc:1067] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:18.962539 17701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35634
I20240502 01:49:18.963124 17701 raft_consensus.cc:482] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:18.963505 17701 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:18.970045 17701 raft_consensus.cc:504] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.972820 17777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:18.973351 17777 raft_consensus.cc:3032] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:18.973654 17777 raft_consensus.cc:729] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:18.974339 17777 consensus_queue.cc:260] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.975266 17777 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:18.972820 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:18.976503 17852 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:18.983119 17777 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:18.983392 17852 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:18.984249 17664 leader_election.cc:302] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:18.984997 17930 raft_consensus.cc:2781] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:18.985391 17930 raft_consensus.cc:686] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:18.986122 17701 leader_election.cc:288] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:18.986160 17930 consensus_queue.cc:237] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:18.988970 17966 consensus_queue.cc:1035] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:18.995033 17593 catalog_manager.cc:5484] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 1 to 2, leader changed from e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 2 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.004307 17966 consensus_queue.cc:1035] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.016498 17930 raft_consensus.cc:979] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: : Instructing follower e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:19.016995 17966 raft_consensus.cc:1067] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Signalling peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:19.018792 17777 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2196a1b679d94d769f5593718e77173b"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:55794
I20240502 01:49:19.019255 17777 raft_consensus.cc:482] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.019549 17777 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.024113 17777 raft_consensus.cc:504] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.025657 17777 leader_election.cc:288] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 2 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.026716 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:19.026800 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.027204 17701 raft_consensus.cc:3032] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.027560 17701 raft_consensus.cc:729] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.028061 17851 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.028229 17701 consensus_queue.cc:260] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.029268 17701 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.034686 17701 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 2.
I20240502 01:49:19.035837 17741 leader_election.cc:302] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:19.036082 17851 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 2.
I20240502 01:49:19.036537 17912 raft_consensus.cc:2781] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.036989 17912 raft_consensus.cc:686] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:19.037602 17912 consensus_queue.cc:237] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.044072 17594 catalog_manager.cc:5484] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 1 to 2, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.078222 17912 consensus_queue.cc:1035] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:19.090237 17959 consensus_queue.cc:1035] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.105268 17966 consensus_queue.cc:1035] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.105633 17928 raft_consensus.cc:979] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.106099 17928 raft_consensus.cc:1067] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.110076 17701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35634
I20240502 01:49:19.110697 17701 raft_consensus.cc:482] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.111081 17701 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.117079 17916 raft_consensus.cc:979] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.117542 17916 raft_consensus.cc:1067] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.117812 17701 raft_consensus.cc:504] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.119408 17700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35646
I20240502 01:49:19.120012 17700 raft_consensus.cc:482] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.120357 17700 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.120543 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:19.120770 17701 leader_election.cc:288] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.121115 17778 raft_consensus.cc:3032] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.121448 17778 raft_consensus.cc:729] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:19.121901 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.122475 17851 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.122175 17778 consensus_queue.cc:260] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.123471 17778 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.130236 17778 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.132088 17664 leader_election.cc:302] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:19.132028 17966 consensus_queue.cc:1035] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:19.133045 17930 raft_consensus.cc:2781] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.132907 17700 raft_consensus.cc:504] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.133638 17930 raft_consensus.cc:686] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.134505 17930 consensus_queue.cc:237] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.137596 17899 raft_consensus.cc:979] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: : Instructing follower e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:19.137701 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.138008 17899 raft_consensus.cc:1067] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Signalling peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:19.138162 17851 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.138294 17852 raft_consensus.cc:3032] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.139055 17852 raft_consensus.cc:729] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:19.139777 17852 consensus_queue.cc:260] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.140884 17852 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.146358 17930 consensus_queue.cc:1035] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.149762 17700 leader_election.cc:288] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.150985 17593 catalog_manager.cc:5484] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 1 to 2, leader changed from e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 2 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.156643 17852 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.158260 17663 leader_election.cc:302] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:19.159150 17930 raft_consensus.cc:2781] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.159603 17930 raft_consensus.cc:686] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.160406 17930 consensus_queue.cc:237] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.165535 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:19.166116 17778 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.173800 17778 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.174590 17903 consensus_queue.cc:1035] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.176082 17775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:55794
I20240502 01:49:19.176685 17775 raft_consensus.cc:482] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.177044 17775 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.174568 17593 catalog_manager.cc:5484] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 1 to 2, leader changed from 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 2 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.182965 17851 raft_consensus.cc:3037] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:19.184381 17775 raft_consensus.cc:504] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.185456 17776 raft_consensus.cc:1226] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Rejecting Update request from peer 25eb7db9986849549a52ca275d122dbb for earlier term 1. Current term is 2. Ops: []
I20240502 01:49:19.190250 17930 consensus_queue.cc:1046] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.190361 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:19.191318 17988 raft_consensus.cc:3032] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.191685 17988 raft_consensus.cc:729] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.192512 17988 consensus_queue.cc:260] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.193987 17988 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.203455 17775 leader_election.cc:288] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 2 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.203665 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.204298 17852 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.210126 17930 consensus_queue.cc:1035] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.217185 17700 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 2.
I20240502 01:49:19.219941 17741 leader_election.cc:302] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:19.220876 17928 raft_consensus.cc:2781] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.221333 17928 raft_consensus.cc:686] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:19.221511 17903 consensus_queue.cc:1035] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:19.222121 17852 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 2.
I20240502 01:49:19.223073 17928 consensus_queue.cc:237] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.232797 17903 raft_consensus.cc:979] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.233242 17903 raft_consensus.cc:1067] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.234131 17988 raft_consensus.cc:979] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: : Instructing follower 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:19.234549 17988 raft_consensus.cc:1067] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Signalling peer 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:19.234973 17701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35646
I20240502 01:49:19.235553 17701 raft_consensus.cc:482] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.235929 17701 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.231539 17594 catalog_manager.cc:5484] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 1 to 2, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.241914 17851 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35342
I20240502 01:49:19.242463 17851 raft_consensus.cc:482] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.242844 17851 raft_consensus.cc:3037] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.242555 17701 raft_consensus.cc:504] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.250263 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:19.250587 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.251009 17701 leader_election.cc:288] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.252020 17852 raft_consensus.cc:3032] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.252594 17852 raft_consensus.cc:729] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:19.253319 17852 consensus_queue.cc:260] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.254173 17852 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.252317 17851 raft_consensus.cc:504] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.256107 17851 leader_election.cc:288] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 2 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:19.257156 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:19.257670 17778 raft_consensus.cc:3037] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.257463 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9467a8fb444f8baaf3349f68bec2cb" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:19.258150 17700 raft_consensus.cc:3032] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.258474 17700 raft_consensus.cc:729] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.259251 17700 consensus_queue.cc:260] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.260393 17700 raft_consensus.cc:3037] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.266086 17852 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.267061 17700 raft_consensus.cc:2454] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 2.
I20240502 01:49:19.267194 17663 leader_election.cc:302] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:19.268131 17899 raft_consensus.cc:2781] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.268599 17899 raft_consensus.cc:686] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.269397 17899 consensus_queue.cc:237] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.252020 17775 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.272256 17778 raft_consensus.cc:2454] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 2.
I20240502 01:49:19.273233 17816 leader_election.cc:302] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:19.277282 17974 raft_consensus.cc:2781] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.277550 17775 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.277793 17974 raft_consensus.cc:686] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 2 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:19.279299 17593 catalog_manager.cc:5484] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 1 to 2, leader changed from 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 2 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.279822 17974 consensus_queue.cc:237] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.290102 17595 catalog_manager.cc:5484] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 1 to 2, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 2 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:19.320245 17916 raft_consensus.cc:979] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.320708 17974 raft_consensus.cc:1067] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:19.322175 17701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e8e64a268154607acf2007372270c14"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35646
I20240502 01:49:19.322777 17701 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.323204 17701 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.330408 17701 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.332625 17701 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.333387 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:19.333565 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.333995 17775 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.334172 17852 raft_consensus.cc:3032] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.334524 17852 raft_consensus.cc:729] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:19.335284 17852 consensus_queue.cc:260] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.336525 17852 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.338721 17775 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.339569 17664 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:19.340216 17899 raft_consensus.cc:2781] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.340579 17899 raft_consensus.cc:686] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:19.341333 17899 consensus_queue.cc:237] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.342706 17852 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:19.347255 17595 catalog_manager.cc:5484] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 1 to 2, leader changed from 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 2 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.358093 17903 raft_consensus.cc:979] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: : Instructing follower e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:19.358614 17974 raft_consensus.cc:1067] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Signalling peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:19.360406 17775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:55804
I20240502 01:49:19.360998 17775 raft_consensus.cc:482] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:19.361340 17775 raft_consensus.cc:3037] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.366158 17775 raft_consensus.cc:504] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.367676 17775 leader_election.cc:288] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 2 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:19.368674 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:19.369249 17701 raft_consensus.cc:3037] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.369081 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8803ce7fc814fc1b7e9058cb542ca3c" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:19.369824 17852 raft_consensus.cc:3032] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:19.370147 17852 raft_consensus.cc:729] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:19.370944 17852 consensus_queue.cc:260] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.372321 17852 raft_consensus.cc:3037] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:19.374461 17701 raft_consensus.cc:2454] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 2.
I20240502 01:49:19.375522 17741 leader_election.cc:302] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:19.376330 17928 raft_consensus.cc:2781] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:19.376788 17928 raft_consensus.cc:686] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:19.377588 17928 consensus_queue.cc:237] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:19.384788 17852 raft_consensus.cc:2454] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 2.
I20240502 01:49:19.387348 17775 raft_consensus.cc:1261] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.390307 17899 consensus_queue.cc:1035] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.394281 17595 catalog_manager.cc:5484] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 1 to 2, leader changed from 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131) to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 2 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:19.399492 17852 raft_consensus.cc:1261] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.401091 17930 consensus_queue.cc:1035] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.473875 17851 raft_consensus.cc:1261] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.475360 17928 consensus_queue.cc:1035] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.483642 17701 raft_consensus.cc:1261] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.485464 17928 consensus_queue.cc:1035] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.664655 17851 raft_consensus.cc:1261] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.666038 17930 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.675395 17775 raft_consensus.cc:1261] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.676831 17988 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.685118 17775 raft_consensus.cc:1261] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.686554 17988 consensus_queue.cc:1035] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.698318 17851 raft_consensus.cc:1261] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.699890 17988 consensus_queue.cc:1035] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.728823 17701 raft_consensus.cc:1261] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.730206 17928 consensus_queue.cc:1035] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.741377 17851 raft_consensus.cc:1261] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.743073 17959 consensus_queue.cc:1035] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.782258 17778 raft_consensus.cc:1261] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.783789 17974 consensus_queue.cc:1035] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.784523 17778 raft_consensus.cc:1261] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.786188 17966 consensus_queue.cc:1035] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.796628 17701 raft_consensus.cc:1261] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.798673 17916 consensus_queue.cc:1035] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.800962 17851 raft_consensus.cc:1261] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.802569 17966 consensus_queue.cc:1035] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.834028 17851 raft_consensus.cc:1261] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.835201 17966 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.844051 17778 raft_consensus.cc:1261] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.845669 17966 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.857353 17851 raft_consensus.cc:1261] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.858753 17959 consensus_queue.cc:1035] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:19.871003 17701 raft_consensus.cc:1261] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:19.875053 17928 consensus_queue.cc:1035] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:21.951349 17046 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240502 01:49:21.952111 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:21.972589 17701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35660
I20240502 01:49:21.973111 17701 raft_consensus.cc:595] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Received request to transfer leadership to TS 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:21.975207 17701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e8e64a268154607acf2007372270c14"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35660
I20240502 01:49:21.975677 17701 raft_consensus.cc:595] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Received request to transfer leadership to TS 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:21.976603 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 01:49:21.977012 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:22.207103 17966 raft_consensus.cc:979] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: : Instructing follower 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:22.207509 17966 raft_consensus.cc:1067] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Signalling peer 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:22.208896 17851 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35342
I20240502 01:49:22.209316 17851 raft_consensus.cc:482] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:22.209550 17851 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:22.213811 17851 raft_consensus.cc:504] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:22.215173 17851 leader_election.cc:288] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049)
I20240502 01:49:22.216226 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:22.216217 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:22.216990 17701 raft_consensus.cc:3032] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:22.217048 17778 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:22.217442 17701 raft_consensus.cc:729] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:22.218223 17701 consensus_queue.cc:260] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:22.219487 17701 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:22.224063 17778 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:22.225293 17817 leader_election.cc:302] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:22.225946 17974 raft_consensus.cc:2781] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:22.226006 17701 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:22.227022 17974 raft_consensus.cc:686] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:22.227809 17974 consensus_queue.cc:237] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:22.236129 17593 catalog_manager.cc:5484] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 2 to 3, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 3 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:22.505725 17966 raft_consensus.cc:979] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: : Instructing follower 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:22.506191 18003 raft_consensus.cc:1067] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Signalling peer 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:22.507580 17851 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e8e64a268154607acf2007372270c14"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35342
I20240502 01:49:22.508070 17851 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:22.508334 17851 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:22.512260 17851 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:22.513571 17851 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:22.514434 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:22.514580 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:22.514940 17701 raft_consensus.cc:3032] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:22.515071 17778 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:22.515233 17701 raft_consensus.cc:729] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:22.515971 17701 consensus_queue.cc:260] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:22.516880 17701 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:22.519747 17778 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:22.520851 17817 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:22.521540 17974 raft_consensus.cc:2781] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:22.521996 17974 raft_consensus.cc:686] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:22.521988 17701 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:22.522889 17974 consensus_queue.cc:237] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:22.529249 17593 catalog_manager.cc:5484] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 2 to 3, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 3 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:22.611315 17701 raft_consensus.cc:1261] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:22.612593 17974 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:22.622463 17778 raft_consensus.cc:1261] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:22.624114 17974 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:22.977763 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:22.983163 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:22.983552 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:23.050357 17778 raft_consensus.cc:1261] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:23.051542 17974 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:23.060516 17701 raft_consensus.cc:1261] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:23.062215 18004 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:23.984392 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:23.989930 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:23.990304 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:24.991159 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:24.996438 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:24.996848 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:26.413834 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:26.420650 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:26.421017 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:49:26.506469 17875 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0de0 after lost signal to thread 17049
W20240502 01:49:26.507912 17875 debug-util.cc:398] Leaking SignalData structure 0x7b080019e1c0 after lost signal to thread 17631
W20240502 01:49:26.509563 17875 debug-util.cc:398] Leaking SignalData structure 0x7b0800194c00 after lost signal to thread 17726
W20240502 01:49:26.510529 17875 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4280 after lost signal to thread 17802
W20240502 01:49:26.511313 17875 debug-util.cc:398] Leaking SignalData structure 0x7b080011f920 after lost signal to thread 17878
I20240502 01:49:27.421809 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:27.427822 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:27.428237 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:28.429004 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:28.434114 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:28.434507 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:29.435252 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:29.441481 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:29.441944 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:30.442711 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:30.447979 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:30.448361 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:31.449139 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:31.454272 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:31.454659 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:32.477255 17046 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240502 01:49:32.481503 17851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35374
I20240502 01:49:32.482115 17851 raft_consensus.cc:595] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:32.492429 17851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e8e64a268154607acf2007372270c14"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35374
I20240502 01:49:32.492837 17851 raft_consensus.cc:595] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:32.496639 17775 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:55816
I20240502 01:49:32.497104 17775 raft_consensus.cc:595] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:32.500952 17775 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2196a1b679d94d769f5593718e77173b"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:55816
I20240502 01:49:32.501366 17775 raft_consensus.cc:595] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:32.505288 17775 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
mode: GRACEFUL
new_leader_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:55816
I20240502 01:49:32.505738 17775 raft_consensus.cc:595] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Received request to transfer leadership to TS 25eb7db9986849549a52ca275d122dbb
I20240502 01:49:32.584745 18039 raft_consensus.cc:979] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.585237 18020 raft_consensus.cc:1067] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.586532 17702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35634
I20240502 01:49:32.587033 17702 raft_consensus.cc:482] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:32.587368 17702 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:32.590886 18039 raft_consensus.cc:979] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.591331 18020 raft_consensus.cc:1067] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.592619 17700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35634
I20240502 01:49:32.593020 17700 raft_consensus.cc:482] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:32.593245 17700 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:32.594118 17702 raft_consensus.cc:504] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.595999 17702 leader_election.cc:288] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:32.596823 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:32.596901 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:32.597854 17775 raft_consensus.cc:3032] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:49:32.597941 17851 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:32.598150 17775 raft_consensus.cc:729] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:32.599030 17775 consensus_queue.cc:260] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.599220 17700 raft_consensus.cc:504] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.600174 17775 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:32.601341 17700 leader_election.cc:288] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 3 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:32.601815 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:32.602452 17776 raft_consensus.cc:3032] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:32.602774 17776 raft_consensus.cc:729] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:32.603045 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e314bdfd1de844dca1ca9bacc6ac5778" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:32.603569 17850 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:32.603547 17776 consensus_queue.cc:260] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.605044 17776 raft_consensus.cc:3037] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:32.606752 17775 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:32.606986 17851 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 2.
I20240502 01:49:32.608070 17664 leader_election.cc:302] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:32.608970 18034 raft_consensus.cc:2781] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:32.609930 18034 raft_consensus.cc:686] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:32.610719 18034 consensus_queue.cc:237] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.611075 17776 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 3.
I20240502 01:49:32.611708 17850 raft_consensus.cc:2454] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 3.
I20240502 01:49:32.612082 17664 leader_election.cc:302] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:32.612946 18040 raft_consensus.cc:2781] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:32.615427 18040 raft_consensus.cc:686] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 3 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:32.616590 18040 consensus_queue.cc:237] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.626201 17593 catalog_manager.cc:5484] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 1 to 2, leader changed from e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 2 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:32.642514 18039 raft_consensus.cc:979] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.642920 18020 raft_consensus.cc:1067] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.642297 17593 catalog_manager.cc:5484] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 2 to 3, leader changed from e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 3 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:32.644196 17700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2196a1b679d94d769f5593718e77173b"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35634
I20240502 01:49:32.644659 17700 raft_consensus.cc:482] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:32.644905 17700 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:32.649611 17700 raft_consensus.cc:504] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.650939 17700 leader_election.cc:288] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 3 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:32.651638 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:32.652209 17776 raft_consensus.cc:3032] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:32.652258 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2196a1b679d94d769f5593718e77173b" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:32.652482 17776 raft_consensus.cc:729] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:32.652789 17850 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:32.653133 17776 consensus_queue.cc:260] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.654057 17776 raft_consensus.cc:3037] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:32.657017 17850 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 3.
I20240502 01:49:32.658035 17663 leader_election.cc:302] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:32.658418 17776 raft_consensus.cc:2454] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 3.
I20240502 01:49:32.658751 18043 raft_consensus.cc:2781] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:32.659142 18043 raft_consensus.cc:686] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 3 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:32.659801 18043 consensus_queue.cc:237] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.666231 17593 catalog_manager.cc:5484] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 2 to 3, leader changed from e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 3 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:32.667927 18037 raft_consensus.cc:979] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.668427 18050 raft_consensus.cc:1067] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.669764 17700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e8e64a268154607acf2007372270c14"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35646
I20240502 01:49:32.670156 17700 raft_consensus.cc:482] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:32.670400 17700 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:32.675326 17700 raft_consensus.cc:504] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.676672 17700 leader_election.cc:288] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 4 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:32.677299 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:32.677579 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8e64a268154607acf2007372270c14" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:32.677829 17776 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:32.678085 17850 raft_consensus.cc:3032] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:49:32.678366 17850 raft_consensus.cc:729] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:32.678970 17850 consensus_queue.cc:260] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.679960 17850 raft_consensus.cc:3037] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:32.682181 17776 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 4.
I20240502 01:49:32.682988 17664 leader_election.cc:302] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:32.683679 18043 raft_consensus.cc:2781] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:49:32.684072 18043 raft_consensus.cc:686] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 4 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:32.684507 17850 raft_consensus.cc:2454] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 4.
I20240502 01:49:32.684729 18043 consensus_queue.cc:237] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.690809 17593 catalog_manager.cc:5484] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 3 to 4, leader changed from 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 4 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:32.712345 18037 raft_consensus.cc:979] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: : Instructing follower 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.712814 18050 raft_consensus.cc:1067] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Signalling peer 25eb7db9986849549a52ca275d122dbb to start an election
I20240502 01:49:32.713982 17700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
from {username='slave'} at 127.0.0.1:35646
I20240502 01:49:32.714375 17700 raft_consensus.cc:482] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:32.714609 17700 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:32.718554 17700 raft_consensus.cc:504] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.719759 17700 leader_election.cc:288] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 4 election: Requested vote from peers e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:32.720463 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:32.720713 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee3808f70484d27b5d18bcbb791e565" candidate_uuid: "25eb7db9986849549a52ca275d122dbb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:32.721067 17776 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:32.721261 17850 raft_consensus.cc:3032] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:49:32.721619 17850 raft_consensus.cc:729] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:32.722393 17850 consensus_queue.cc:260] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.723711 17850 raft_consensus.cc:3037] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:49:32.727766 17776 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 4.
I20240502 01:49:32.728842 17664 leader_election.cc:302] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:32.729605 18043 raft_consensus.cc:2781] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:49:32.730062 18043 raft_consensus.cc:686] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 4 LEADER]: Becoming Leader. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:32.730118 17850 raft_consensus.cc:2454] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25eb7db9986849549a52ca275d122dbb in term 4.
I20240502 01:49:32.730970 18043 consensus_queue.cc:237] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:32.738097 17593 catalog_manager.cc:5484] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb reported cstate change: term changed from 3 to 4, leader changed from 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131) to 25eb7db9986849549a52ca275d122dbb (127.16.165.129). New cstate: current_term: 4 leader_uuid: "25eb7db9986849549a52ca275d122dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:33.012753 17775 raft_consensus.cc:1261] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:33.013799 18043 consensus_queue.cc:1035] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:33.023351 17850 raft_consensus.cc:1261] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:33.026022 18042 consensus_queue.cc:1035] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:33.049896 17850 raft_consensus.cc:1261] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:33.050932 18042 consensus_queue.cc:1035] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:33.059186 17775 raft_consensus.cc:1261] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:49:33.060755 18040 consensus_queue.cc:1035] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:49:33.111256 17775 raft_consensus.cc:1261] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:33.112365 18040 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:33.121781 17850 raft_consensus.cc:1261] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:33.123333 18042 consensus_queue.cc:1035] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:33.161970 17775 raft_consensus.cc:1261] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:33.162962 18061 consensus_queue.cc:1035] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:33.170990 17850 raft_consensus.cc:1261] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:33.172806 18040 consensus_queue.cc:1035] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:33.231737 17850 raft_consensus.cc:1261] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:33.233038 18040 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:33.240926 17775 raft_consensus.cc:1261] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Refusing update from remote peer 25eb7db9986849549a52ca275d122dbb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 01:49:33.242362 18042 consensus_queue.cc:1035] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:35.536461 17046 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240502 01:49:35.537077 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:35.544732 17700 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:35660
I20240502 01:49:35.545203 17700 raft_consensus.cc:595] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Received request to transfer leadership to TS e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:35.547624 17700 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35660
I20240502 01:49:35.548095 17700 raft_consensus.cc:595] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Received request to transfer leadership to TS 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:35.550209 17700 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:35660
I20240502 01:49:35.550678 17700 raft_consensus.cc:595] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Received request to transfer leadership to TS e38f7bf0cc1e4318bd5b1ecb6c6beb05
I20240502 01:49:35.552788 17700 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f"
dest_uuid: "25eb7db9986849549a52ca275d122dbb"
mode: GRACEFUL
new_leader_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35660
I20240502 01:49:35.553206 17700 raft_consensus.cc:595] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Received request to transfer leadership to TS 8c03ff01971e4877865c9bfa32d01da8
I20240502 01:49:35.554100 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 01:49:35.554529 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:35.731674 18067 raft_consensus.cc:979] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: : Instructing follower 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:35.732439 18068 raft_consensus.cc:1067] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Signalling peer 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:35.734134 17850 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35342
I20240502 01:49:35.734768 17850 raft_consensus.cc:482] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:35.735126 17850 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.741811 17850 raft_consensus.cc:504] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.744371 17850 leader_election.cc:288] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:35.745528 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:35.745801 17776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9db6c971444af2a9fdc56330f4cf4f" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:35.746244 17700 raft_consensus.cc:3032] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:35.746361 17776 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.746523 17700 raft_consensus.cc:729] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:35.747346 17700 consensus_queue.cc:260] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.748574 17700 raft_consensus.cc:3037] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.754105 17776 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:35.756126 17817 leader_election.cc:302] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:35.757102 18068 raft_consensus.cc:979] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: : Instructing follower e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:35.757405 17700 raft_consensus.cc:2454] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:35.757537 18050 raft_consensus.cc:2781] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:35.757925 18040 raft_consensus.cc:1067] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Signalling peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:35.759804 18050 raft_consensus.cc:686] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:35.760075 17775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:55794
I20240502 01:49:35.760708 17775 raft_consensus.cc:482] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:35.761005 17775 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.761189 18050 consensus_queue.cc:237] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.767055 17775 raft_consensus.cc:504] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.769750 17775 leader_election.cc:288] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 3 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:35.771615 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:35.771680 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76843eda8e4429cbe422ae6cb3f0b45" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:35.772452 17851 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.771366 17594 catalog_manager.cc:5484] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 2 to 3, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 3 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:35.773149 17700 raft_consensus.cc:3032] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:35.773494 17700 raft_consensus.cc:729] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:35.774322 17700 consensus_queue.cc:260] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.775950 17700 raft_consensus.cc:3037] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.779278 17851 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 3.
I20240502 01:49:35.780968 17741 leader_election.cc:302] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:35.781251 17700 raft_consensus.cc:2454] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 3.
I20240502 01:49:35.782114 18020 raft_consensus.cc:2781] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:35.783327 18020 raft_consensus.cc:686] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:35.784334 18020 consensus_queue.cc:237] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.792651 17593 catalog_manager.cc:5484] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 2 to 3, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 3 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:35.876329 18068 raft_consensus.cc:979] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: : Instructing follower e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:35.876819 18067 raft_consensus.cc:1067] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Signalling peer e38f7bf0cc1e4318bd5b1ecb6c6beb05 to start an election
I20240502 01:49:35.878337 17775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611"
dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
from {username='slave'} at 127.0.0.1:55794
I20240502 01:49:35.878782 17775 raft_consensus.cc:482] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:35.879082 17775 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.884100 17775 raft_consensus.cc:504] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.885110 18068 raft_consensus.cc:979] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: : Instructing follower 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:35.885592 18067 raft_consensus.cc:1067] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Signalling peer 8c03ff01971e4877865c9bfa32d01da8 to start an election
I20240502 01:49:35.885715 17775 leader_election.cc:288] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 3 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131:34205)
I20240502 01:49:35.886437 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:35.887071 17700 raft_consensus.cc:3032] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:35.886744 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425497d573154f9195ad16f6d2ee5611" candidate_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
I20240502 01:49:35.887351 17700 raft_consensus.cc:729] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:35.887121 17850 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2"
dest_uuid: "8c03ff01971e4877865c9bfa32d01da8"
from {username='slave'} at 127.0.0.1:35342
I20240502 01:49:35.887735 17851 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.887878 17850 raft_consensus.cc:482] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:49:35.888366 17850 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.888166 17700 consensus_queue.cc:260] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.889483 17700 raft_consensus.cc:3037] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.892182 17851 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 3.
I20240502 01:49:35.893067 17741 leader_election.cc:302] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c03ff01971e4877865c9bfa32d01da8, e38f7bf0cc1e4318bd5b1ecb6c6beb05; no voters:
I20240502 01:49:35.893682 18020 raft_consensus.cc:2781] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:35.894018 18020 raft_consensus.cc:686] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 LEADER]: Becoming Leader. State: Replica: e38f7bf0cc1e4318bd5b1ecb6c6beb05, State: Running, Role: LEADER
I20240502 01:49:35.894912 18020 consensus_queue.cc:237] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.895120 17850 raft_consensus.cc:504] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.895612 17700 raft_consensus.cc:2454] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38f7bf0cc1e4318bd5b1ecb6c6beb05 in term 3.
I20240502 01:49:35.897331 17850 leader_election.cc:288] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Requested vote from peers 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049), e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130:36169)
I20240502 01:49:35.897961 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "25eb7db9986849549a52ca275d122dbb"
I20240502 01:49:35.898491 17700 raft_consensus.cc:3032] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:49:35.898352 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529aed8c4c14bdd870a4d23d857baf2" candidate_uuid: "8c03ff01971e4877865c9bfa32d01da8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05"
I20240502 01:49:35.898766 17700 raft_consensus.cc:729] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25eb7db9986849549a52ca275d122dbb, State: Running, Role: LEADER
I20240502 01:49:35.899127 17775 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.899392 17700 consensus_queue.cc:260] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.900416 17700 raft_consensus.cc:3037] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:49:35.904665 17700 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:35.905855 17816 leader_election.cc:302] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eb7db9986849549a52ca275d122dbb, 8c03ff01971e4877865c9bfa32d01da8; no voters:
I20240502 01:49:35.906704 18050 raft_consensus.cc:2781] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:49:35.907207 18050 raft_consensus.cc:686] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Becoming Leader. State: Replica: 8c03ff01971e4877865c9bfa32d01da8, State: Running, Role: LEADER
I20240502 01:49:35.907851 17775 raft_consensus.cc:2454] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c03ff01971e4877865c9bfa32d01da8 in term 3.
I20240502 01:49:35.908015 18050 consensus_queue.cc:237] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } }
I20240502 01:49:35.910743 17593 catalog_manager.cc:5484] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 reported cstate change: term changed from 2 to 3, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to e38f7bf0cc1e4318bd5b1ecb6c6beb05 (127.16.165.130). New cstate: current_term: 3 leader_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:35.916776 17594 catalog_manager.cc:5484] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 reported cstate change: term changed from 2 to 3, leader changed from 25eb7db9986849549a52ca275d122dbb (127.16.165.129) to 8c03ff01971e4877865c9bfa32d01da8 (127.16.165.131). New cstate: current_term: 3 leader_uuid: "8c03ff01971e4877865c9bfa32d01da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:36.192420 17775 raft_consensus.cc:1261] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.193526 18050 consensus_queue.cc:1035] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:36.207734 17700 raft_consensus.cc:1261] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.209475 18084 consensus_queue.cc:1035] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:36.295975 17850 raft_consensus.cc:1261] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.297045 18020 consensus_queue.cc:1035] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:36.305048 17700 raft_consensus.cc:1261] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.306612 18020 consensus_queue.cc:1035] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:36.425191 17850 raft_consensus.cc:1261] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.426301 18088 consensus_queue.cc:1035] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c03ff01971e4877865c9bfa32d01da8" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:36.434607 17700 raft_consensus.cc:1261] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Refusing update from remote peer e38f7bf0cc1e4318bd5b1ecb6c6beb05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.436882 18075 consensus_queue.cc:1035] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:36.476166 17775 raft_consensus.cc:1261] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.477255 18084 consensus_queue.cc:1035] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38f7bf0cc1e4318bd5b1ecb6c6beb05" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:36.484400 17700 raft_consensus.cc:1261] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Refusing update from remote peer 8c03ff01971e4877865c9bfa32d01da8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:49:36.485852 18084 consensus_queue.cc:1035] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7db9986849549a52ca275d122dbb" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:36.555476 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:36.561543 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:36.562044 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:37.562938 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:37.568375 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:37.568806 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:38.569690 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:38.575702 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:38.576124 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:39.576860 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:39.582437 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:39.582839 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:40.583678 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:40.589634 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:40.590073 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:41.590852 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:41.596415 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:41.596861 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:42.597616 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:42.603646 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:42.604185 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:43.605017 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:43.610149 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:43.610533 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:44.611238 17046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:44.616439 17046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:44.616822 17046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:45.660807 17046 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240502 01:49:45.661458 17046 tablet_server.cc:178] TabletServer@127.16.165.129:0 shutting down...
I20240502 01:49:45.685472 17046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:45.686422 17046 tablet_replica.cc:330] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.688107 17046 raft_consensus.cc:2229] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.689823 17046 raft_consensus.cc:2258] T e314bdfd1de844dca1ca9bacc6ac5778 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.692209 17046 tablet_replica.cc:330] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.692797 17046 raft_consensus.cc:2229] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.693310 17046 raft_consensus.cc:2258] T db9db6c971444af2a9fdc56330f4cf4f P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.696040 17046 tablet_replica.cc:330] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.696813 17046 raft_consensus.cc:2229] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus shutting down.
W20240502 01:49:45.697268 17816 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20240502 01:49:45.697438 17046 raft_consensus.cc:2258] T 8529aed8c4c14bdd870a4d23d857baf2 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.700242 17046 tablet_replica.cc:330] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.700965 17046 raft_consensus.cc:2229] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Raft consensus shutting down.
W20240502 01:49:45.701311 17816 consensus_peers.cc:485] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 -> Peer 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049): Couldn't send request to peer 25eb7db9986849549a52ca275d122dbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:45.701922 17046 raft_consensus.cc:2258] T e8803ce7fc814fc1b7e9058cb542ca3c P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.704576 17046 tablet_replica.cc:330] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.705092 17046 raft_consensus.cc:2229] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 4 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.706269 17046 raft_consensus.cc:2258] T 1ee3808f70484d27b5d18bcbb791e565 P 25eb7db9986849549a52ca275d122dbb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.708156 17046 tablet_replica.cc:330] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.708657 17046 raft_consensus.cc:2229] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.709695 17046 raft_consensus.cc:2258] T 2196a1b679d94d769f5593718e77173b P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.711496 17046 tablet_replica.cc:330] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.712160 17046 raft_consensus.cc:2229] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.712706 17046 raft_consensus.cc:2258] T 425497d573154f9195ad16f6d2ee5611 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.715026 17046 tablet_replica.cc:330] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.715538 17046 raft_consensus.cc:2229] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.716240 17046 raft_consensus.cc:2258] T 6a9467a8fb444f8baaf3349f68bec2cb P 25eb7db9986849549a52ca275d122dbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.718201 17046 tablet_replica.cc:330] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.718740 17046 raft_consensus.cc:2229] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 4 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.720000 17046 raft_consensus.cc:2258] T 9e8e64a268154607acf2007372270c14 P 25eb7db9986849549a52ca275d122dbb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.722084 17046 tablet_replica.cc:330] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb: stopping tablet replica
I20240502 01:49:45.722613 17046 raft_consensus.cc:2229] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.723145 17046 raft_consensus.cc:2258] T b76843eda8e4429cbe422ae6cb3f0b45 P 25eb7db9986849549a52ca275d122dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.766495 17046 tablet_server.cc:194] TabletServer@127.16.165.129:0 shutdown complete.
I20240502 01:49:45.785456 17046 tablet_server.cc:178] TabletServer@127.16.165.130:0 shutting down...
I20240502 01:49:45.812208 17046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:45.813061 17046 tablet_replica.cc:330] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.814260 17046 raft_consensus.cc:2229] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.814797 17046 raft_consensus.cc:2258] T e314bdfd1de844dca1ca9bacc6ac5778 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.817163 17046 tablet_replica.cc:330] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.817734 17046 raft_consensus.cc:2229] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.818809 17046 raft_consensus.cc:2258] T e8803ce7fc814fc1b7e9058cb542ca3c P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.820814 17046 tablet_replica.cc:330] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.821419 17046 raft_consensus.cc:2229] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.821894 17046 raft_consensus.cc:2258] T db9db6c971444af2a9fdc56330f4cf4f P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.823770 17046 tablet_replica.cc:330] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.824443 17046 raft_consensus.cc:2229] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.824967 17046 raft_consensus.cc:2258] T 8529aed8c4c14bdd870a4d23d857baf2 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.826673 17046 tablet_replica.cc:330] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.827137 17046 raft_consensus.cc:2229] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.828138 17046 raft_consensus.cc:2258] T b76843eda8e4429cbe422ae6cb3f0b45 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.829982 17046 tablet_replica.cc:330] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.830417 17046 raft_consensus.cc:2229] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.830861 17046 raft_consensus.cc:2258] T 6a9467a8fb444f8baaf3349f68bec2cb P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.832592 17046 tablet_replica.cc:330] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.833020 17046 raft_consensus.cc:2229] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.833499 17046 raft_consensus.cc:2258] T 9e8e64a268154607acf2007372270c14 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.836047 17046 tablet_replica.cc:330] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.836665 17046 raft_consensus.cc:2229] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.837239 17046 raft_consensus.cc:2258] T 2196a1b679d94d769f5593718e77173b P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 01:49:45.839278 17816 consensus_peers.cc:485] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 -> Peer 25eb7db9986849549a52ca275d122dbb (127.16.165.129:37049): Couldn't send request to peer 25eb7db9986849549a52ca275d122dbb. Status: Network error: Client connection negotiation failed: client connection to 127.16.165.129:37049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:45.839696 17046 tablet_replica.cc:330] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.840246 17046 raft_consensus.cc:2229] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.840677 17046 raft_consensus.cc:2258] T 1ee3808f70484d27b5d18bcbb791e565 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.842381 17046 tablet_replica.cc:330] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05: stopping tablet replica
I20240502 01:49:45.842797 17046 raft_consensus.cc:2229] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.843663 17046 raft_consensus.cc:2258] T 425497d573154f9195ad16f6d2ee5611 P e38f7bf0cc1e4318bd5b1ecb6c6beb05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.872846 17046 tablet_server.cc:194] TabletServer@127.16.165.130:0 shutdown complete.
I20240502 01:49:45.892010 17046 tablet_server.cc:178] TabletServer@127.16.165.131:0 shutting down...
I20240502 01:49:45.916982 17046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:45.917848 17046 tablet_replica.cc:330] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.918735 17046 raft_consensus.cc:2229] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.919770 17046 raft_consensus.cc:2258] T db9db6c971444af2a9fdc56330f4cf4f P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.921627 17046 tablet_replica.cc:330] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.922194 17046 raft_consensus.cc:2229] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.923038 17046 raft_consensus.cc:2258] T 8529aed8c4c14bdd870a4d23d857baf2 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.924865 17046 tablet_replica.cc:330] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.925360 17046 raft_consensus.cc:2229] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.925777 17046 raft_consensus.cc:2258] T e314bdfd1de844dca1ca9bacc6ac5778 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.927505 17046 tablet_replica.cc:330] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.928092 17046 raft_consensus.cc:2229] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.928548 17046 raft_consensus.cc:2258] T e8803ce7fc814fc1b7e9058cb542ca3c P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.930198 17046 tablet_replica.cc:330] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.930702 17046 raft_consensus.cc:2229] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.931157 17046 raft_consensus.cc:2258] T 1ee3808f70484d27b5d18bcbb791e565 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.932842 17046 tablet_replica.cc:330] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.933308 17046 raft_consensus.cc:2229] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.933786 17046 raft_consensus.cc:2258] T 2196a1b679d94d769f5593718e77173b P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.935420 17046 tablet_replica.cc:330] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.935907 17046 raft_consensus.cc:2229] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.936369 17046 raft_consensus.cc:2258] T 425497d573154f9195ad16f6d2ee5611 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.938179 17046 tablet_replica.cc:330] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.938666 17046 raft_consensus.cc:2229] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:49:45.939635 17046 raft_consensus.cc:2258] T 6a9467a8fb444f8baaf3349f68bec2cb P 8c03ff01971e4877865c9bfa32d01da8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.941505 17046 tablet_replica.cc:330] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.941931 17046 raft_consensus.cc:2229] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.942435 17046 raft_consensus.cc:2258] T b76843eda8e4429cbe422ae6cb3f0b45 P 8c03ff01971e4877865c9bfa32d01da8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.944001 17046 tablet_replica.cc:330] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8: stopping tablet replica
I20240502 01:49:45.944442 17046 raft_consensus.cc:2229] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:45.944890 17046 raft_consensus.cc:2258] T 9e8e64a268154607acf2007372270c14 P 8c03ff01971e4877865c9bfa32d01da8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:45.987310 17046 tablet_server.cc:194] TabletServer@127.16.165.131:0 shutdown complete.
I20240502 01:49:46.006145 17046 master.cc:537] Master@127.16.165.190:35283 shutting down...
I20240502 01:49:46.027292 17046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:46.027854 17046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:46.028204 17046 tablet_replica.cc:330] T 00000000000000000000000000000000 P dd1d760441cb46eda6176b35bb949d58: stopping tablet replica
I20240502 01:49:46.046437 17046 master.cc:552] Master@127.16.165.190:35283 shutdown complete.
25eb7db9986849549a52ca275d122dbb 5
8c03ff01971e4877865c9bfa32d01da8 1
e38f7bf0cc1e4318bd5b1ecb6c6beb05 4
25eb7db9986849549a52ca275d122dbb 3
8c03ff01971e4877865c9bfa32d01da8 3
e38f7bf0cc1e4318bd5b1ecb6c6beb05 4
25eb7db9986849549a52ca275d122dbb 8
8c03ff01971e4877865c9bfa32d01da8 1
e38f7bf0cc1e4318bd5b1ecb6c6beb05 1
25eb7db9986849549a52ca275d122dbb 4
8c03ff01971e4877865c9bfa32d01da8 3
e38f7bf0cc1e4318bd5b1ecb6c6beb05 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29723 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240502 01:49:46.084807 17046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.165.190:35883
I20240502 01:49:46.085872 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:46.091903 18107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.092370 18108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.092744 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:46.093339 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:46.094272 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:46.094457 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:46.094594 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614586094578 us; error 0 us; skew 500 ppm
I20240502 01:49:46.095105 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:46.103338 17046 webserver.cc:460] Webserver started at http://127.16.165.190:37775/ using document root <none> and password file <none>
I20240502 01:49:46.103818 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:46.104040 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:46.104315 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:46.105521 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/master-0-root/instance:
uuid: "5547c70b69f349339de8325bac47a8df"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.110050 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:49:46.113266 18115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.113974 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:49:46.114236 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/master-0-root
uuid: "5547c70b69f349339de8325bac47a8df"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.114508 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:46.139619 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:46.140895 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:46.178046 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.190:35883
I20240502 01:49:46.178169 18169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.190:35883 every 8 connection(s)
I20240502 01:49:46.181856 18170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.192020 18170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df: Bootstrap starting.
I20240502 01:49:46.196324 18170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:46.200654 18170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df: No bootstrap required, opened a new log
I20240502 01:49:46.202769 18170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547c70b69f349339de8325bac47a8df" member_type: VOTER }
I20240502 01:49:46.203271 18170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:46.203512 18170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5547c70b69f349339de8325bac47a8df, State: Initialized, Role: FOLLOWER
I20240502 01:49:46.204089 18170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547c70b69f349339de8325bac47a8df" member_type: VOTER }
I20240502 01:49:46.204568 18170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:49:46.204807 18170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:49:46.205085 18170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:46.209221 18170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547c70b69f349339de8325bac47a8df" member_type: VOTER }
I20240502 01:49:46.209757 18170 leader_election.cc:302] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5547c70b69f349339de8325bac47a8df; no voters:
I20240502 01:49:46.210875 18170 leader_election.cc:288] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:49:46.211272 18173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:46.212530 18173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [term 1 LEADER]: Becoming Leader. State: Replica: 5547c70b69f349339de8325bac47a8df, State: Running, Role: LEADER
I20240502 01:49:46.213286 18173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547c70b69f349339de8325bac47a8df" member_type: VOTER }
I20240502 01:49:46.213807 18170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:49:46.218072 18175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5547c70b69f349339de8325bac47a8df. Latest consensus state: current_term: 1 leader_uuid: "5547c70b69f349339de8325bac47a8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547c70b69f349339de8325bac47a8df" member_type: VOTER } }
I20240502 01:49:46.218391 18174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5547c70b69f349339de8325bac47a8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547c70b69f349339de8325bac47a8df" member_type: VOTER } }
I20240502 01:49:46.218914 18175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:46.219044 18174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:46.221974 18180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:49:46.227213 18180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:49:46.229956 17046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:49:46.236193 18180 catalog_manager.cc:1347] Generated new cluster ID: b1f302382d614308b734565adc64802d
I20240502 01:49:46.236521 18180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:49:46.250901 18180 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:49:46.252226 18180 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:49:46.268127 18180 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5547c70b69f349339de8325bac47a8df: Generated new TSK 0
I20240502 01:49:46.268782 18180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:49:46.297196 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:46.303997 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.304545 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.305722 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:46.306313 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:46.307193 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:46.307392 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:46.307541 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614586307522 us; error 0 us; skew 500 ppm
I20240502 01:49:46.308147 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:46.310461 17046 webserver.cc:460] Webserver started at http://127.16.165.129:39691/ using document root <none> and password file <none>
I20240502 01:49:46.310931 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:46.311107 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:46.311352 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:46.312582 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root/instance:
uuid: "26c26643de084d6b8acaff73fa67f794"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.317332 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:46.320672 18199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.321409 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:46.321683 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root
uuid: "26c26643de084d6b8acaff73fa67f794"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.321980 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:46.354269 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:46.355485 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:46.356998 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:46.359246 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:46.359449 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.359663 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:46.359833 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.398700 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.129:38913
I20240502 01:49:46.398808 18264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.129:38913 every 8 connection(s)
I20240502 01:49:46.403684 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:46.410702 18269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.411442 18270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.413426 18272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:46.414072 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:46.415159 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:46.415414 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:46.415611 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614586415592 us; error 0 us; skew 500 ppm
I20240502 01:49:46.416378 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:46.417721 18265 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35883
I20240502 01:49:46.418103 18265 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:46.418852 18265 heartbeater.cc:511] Master 127.16.165.190:35883 requested a full tablet report, sending...
I20240502 01:49:46.419060 17046 webserver.cc:460] Webserver started at http://127.16.165.130:36235/ using document root <none> and password file <none>
I20240502 01:49:46.419555 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:46.419734 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:46.419993 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:46.421106 18133 ts_manager.cc:194] Registered new tserver with Master: 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:49:46.421430 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root/instance:
uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.423451 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37644
I20240502 01:49:46.426759 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240502 01:49:46.430073 18277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.430795 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:49:46.431056 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root
uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.431324 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:46.449453 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:46.450642 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:46.452054 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:46.454553 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:46.454735 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.454960 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:46.455099 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.493820 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.130:33629
I20240502 01:49:46.493917 18341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.130:33629 every 8 connection(s)
I20240502 01:49:46.498240 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:46.505919 18345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.506040 18346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:46.508036 18348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:46.508699 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:46.509577 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:46.509763 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:46.509914 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614586509898 us; error 0 us; skew 500 ppm
I20240502 01:49:46.510246 18342 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35883
I20240502 01:49:46.510519 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:46.510564 18342 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:46.511466 18342 heartbeater.cc:511] Master 127.16.165.190:35883 requested a full tablet report, sending...
I20240502 01:49:46.513310 17046 webserver.cc:460] Webserver started at http://127.16.165.131:38897/ using document root <none> and password file <none>
I20240502 01:49:46.513789 18133 ts_manager.cc:194] Registered new tserver with Master: 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:46.513937 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:46.514232 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:46.514564 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:46.515579 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37656
I20240502 01:49:46.516074 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root/instance:
uuid: "d6698cd7a2d740368c0c89233e16f04c"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.520386 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:49:46.523430 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.524173 17046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:49:46.524459 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root
uuid: "d6698cd7a2d740368c0c89233e16f04c"
format_stamp: "Formatted at 2024-05-02 01:49:46 on dist-test-slave-kj4b"
I20240502 01:49:46.524781 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:46.542690 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:46.543824 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:46.545238 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:46.547320 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:46.547511 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:46.547735 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:46.547900 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 01:49:46.585063 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.131:45775
I20240502 01:49:46.585166 18417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.131:45775 every 8 connection(s)
I20240502 01:49:46.597705 18418 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35883
I20240502 01:49:46.598049 18418 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:46.598726 18418 heartbeater.cc:511] Master 127.16.165.190:35883 requested a full tablet report, sending...
I20240502 01:49:46.600533 18133 ts_manager.cc:194] Registered new tserver with Master: d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:46.601533 17046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013492936s
I20240502 01:49:46.602034 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37666
I20240502 01:49:46.604332 17046 test_util.cc:268] Using random seed: -782468340
I20240502 01:49:46.627194 18133 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37676:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:49:46.633191 18133 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:49:46.853988 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 9a75f4d1c0c746f3ba51960f4f4e4f55 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 01:49:46.855443 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a75f4d1c0c746f3ba51960f4f4e4f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.856914 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 47fdfbadfdb3472782d6a189ffbbe112 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 01:49:46.857357 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa7efd1cfbd4c81bbcde777d6eeb082 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 01:49:46.858312 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fdfbadfdb3472782d6a189ffbbe112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.858681 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa7efd1cfbd4c81bbcde777d6eeb082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.859954 18382 tablet_service.cc:1467] Processing CreateTablet for tablet 9a75f4d1c0c746f3ba51960f4f4e4f55 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 01:49:46.861188 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a75f4d1c0c746f3ba51960f4f4e4f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.860998 18304 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa7efd1cfbd4c81bbcde777d6eeb082 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 01:49:46.862237 18376 tablet_service.cc:1467] Processing CreateTablet for tablet da54a35564ce4389a2395ea0832a968b (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 01:49:46.863493 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da54a35564ce4389a2395ea0832a968b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.864997 18375 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4a06b4fbd949e490a4e206fa313777 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 01:49:46.866240 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4a06b4fbd949e490a4e206fa313777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.866132 18300 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4a06b4fbd949e490a4e206fa313777 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 01:49:46.869012 18306 tablet_service.cc:1467] Processing CreateTablet for tablet 9a75f4d1c0c746f3ba51960f4f4e4f55 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 01:49:46.870093 18373 tablet_service.cc:1467] Processing CreateTablet for tablet c3eec00736204e3fa0dcf0797f7b0d75 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 01:49:46.871253 18302 tablet_service.cc:1467] Processing CreateTablet for tablet b7b8a888ea634dcb802badc6a669ecc7 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 01:49:46.872489 18372 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba72fe24abe4fe893e60d48e6679580 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 01:49:46.873003 18299 tablet_service.cc:1467] Processing CreateTablet for tablet caecb89696f549ac8c4ab781e42472b1 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 01:49:46.874296 18303 tablet_service.cc:1467] Processing CreateTablet for tablet af7889fa2a16487d8b767d51bb21abf2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 01:49:46.874660 18377 tablet_service.cc:1467] Processing CreateTablet for tablet b7b8a888ea634dcb802badc6a669ecc7 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 01:49:46.861011 18378 tablet_service.cc:1467] Processing CreateTablet for tablet af7889fa2a16487d8b767d51bb21abf2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 01:49:46.869015 18374 tablet_service.cc:1467] Processing CreateTablet for tablet caecb89696f549ac8c4ab781e42472b1 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 01:49:46.862258 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa7efd1cfbd4c81bbcde777d6eeb082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.877600 18303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7889fa2a16487d8b767d51bb21abf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.869014 18381 tablet_service.cc:1467] Processing CreateTablet for tablet 47fdfbadfdb3472782d6a189ffbbe112 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 01:49:46.878691 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4a06b4fbd949e490a4e206fa313777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.864523 18301 tablet_service.cc:1467] Processing CreateTablet for tablet da54a35564ce4389a2395ea0832a968b (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 01:49:46.882336 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da54a35564ce4389a2395ea0832a968b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.884296 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b8a888ea634dcb802badc6a669ecc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.886240 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caecb89696f549ac8c4ab781e42472b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.889567 18298 tablet_service.cc:1467] Processing CreateTablet for tablet c3eec00736204e3fa0dcf0797f7b0d75 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 01:49:46.890923 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3eec00736204e3fa0dcf0797f7b0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.893630 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba72fe24abe4fe893e60d48e6679580 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 01:49:46.894989 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba72fe24abe4fe893e60d48e6679580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.897063 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a75f4d1c0c746f3ba51960f4f4e4f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.903939 18228 tablet_service.cc:1467] Processing CreateTablet for tablet 47fdfbadfdb3472782d6a189ffbbe112 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 01:49:46.905386 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fdfbadfdb3472782d6a189ffbbe112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.871306 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3eec00736204e3fa0dcf0797f7b0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.906608 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba72fe24abe4fe893e60d48e6679580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.908270 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba72fe24abe4fe893e60d48e6679580 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 01:49:46.908494 18227 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa7efd1cfbd4c81bbcde777d6eeb082 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 01:49:46.909834 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba72fe24abe4fe893e60d48e6679580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.910382 18226 tablet_service.cc:1467] Processing CreateTablet for tablet af7889fa2a16487d8b767d51bb21abf2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 01:49:46.911542 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7889fa2a16487d8b767d51bb21abf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.912377 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caecb89696f549ac8c4ab781e42472b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.913652 18224 tablet_service.cc:1467] Processing CreateTablet for tablet da54a35564ce4389a2395ea0832a968b (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 01:49:46.915573 18222 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4a06b4fbd949e490a4e206fa313777 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 01:49:46.916841 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fdfbadfdb3472782d6a189ffbbe112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.917412 18221 tablet_service.cc:1467] Processing CreateTablet for tablet caecb89696f549ac8c4ab781e42472b1 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 01:49:46.919454 18220 tablet_service.cc:1467] Processing CreateTablet for tablet c3eec00736204e3fa0dcf0797f7b0d75 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 01:49:46.921707 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7889fa2a16487d8b767d51bb21abf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.912330 18225 tablet_service.cc:1467] Processing CreateTablet for tablet b7b8a888ea634dcb802badc6a669ecc7 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 01:49:46.922600 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b8a888ea634dcb802badc6a669ecc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.924969 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b8a888ea634dcb802badc6a669ecc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.928519 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caecb89696f549ac8c4ab781e42472b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.931234 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4a06b4fbd949e490a4e206fa313777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.931905 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3eec00736204e3fa0dcf0797f7b0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.936245 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da54a35564ce4389a2395ea0832a968b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.940069 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa7efd1cfbd4c81bbcde777d6eeb082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.993794 18300 tablet_service.cc:1467] Processing CreateTablet for tablet ffa5663e2ff0414b948af8d5967266cc (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 01:49:46.994138 18372 tablet_service.cc:1467] Processing CreateTablet for tablet cf23c8fedfe4430bbb092f9e927d55e5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 01:49:46.995126 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa5663e2ff0414b948af8d5967266cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:46.995584 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf23c8fedfe4430bbb092f9e927d55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.003029 18301 tablet_service.cc:1467] Processing CreateTablet for tablet cf23c8fedfe4430bbb092f9e927d55e5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 01:49:47.025525 18382 tablet_service.cc:1467] Processing CreateTablet for tablet ffa5663e2ff0414b948af8d5967266cc (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 01:49:47.030483 18300 tablet_service.cc:1467] Processing CreateTablet for tablet 231a18b76f334dc281187e58de8d90e4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 01:49:47.031009 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf23c8fedfe4430bbb092f9e927d55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.030472 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa5663e2ff0414b948af8d5967266cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.034875 18298 tablet_service.cc:1467] Processing CreateTablet for tablet e3116ed5b5254b4d926710cf1e5ed93a (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 01:49:47.038061 18433 tablet_bootstrap.cc:492] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:47.037694 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 231a18b76f334dc281187e58de8d90e4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 01:49:47.039147 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231a18b76f334dc281187e58de8d90e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.042672 18304 tablet_service.cc:1467] Processing CreateTablet for tablet 81a9d50e756d4871a8789ea75c671cca (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 01:49:47.050849 18306 tablet_service.cc:1467] Processing CreateTablet for tablet 7943deee0b2f49cf8a391e59d9676aa1 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 01:49:47.051427 18375 tablet_service.cc:1467] Processing CreateTablet for tablet e3116ed5b5254b4d926710cf1e5ed93a (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 01:49:47.040421 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 5090d855c6864a3d904fe1141d503d9d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 01:49:47.068850 18435 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:47.072901 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 231a18b76f334dc281187e58de8d90e4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 01:49:47.055563 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a9d50e756d4871a8789ea75c671cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.074120 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231a18b76f334dc281187e58de8d90e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.076838 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3116ed5b5254b4d926710cf1e5ed93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.078713 18434 tablet_bootstrap.cc:492] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:47.079392 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7943deee0b2f49cf8a391e59d9676aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.079536 18226 tablet_service.cc:1467] Processing CreateTablet for tablet ffa5663e2ff0414b948af8d5967266cc (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 01:49:47.074164 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231a18b76f334dc281187e58de8d90e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.065840 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3116ed5b5254b4d926710cf1e5ed93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.084057 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 4e46b9ec3add4df0b2026081227bbcd2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 01:49:47.096671 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e46b9ec3add4df0b2026081227bbcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.090555 18433 tablet_bootstrap.cc:654] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.095301 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 5090d855c6864a3d904fe1141d503d9d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 01:49:47.111231 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5090d855c6864a3d904fe1141d503d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.112344 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5090d855c6864a3d904fe1141d503d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.102097 18222 tablet_service.cc:1467] Processing CreateTablet for tablet 81a9d50e756d4871a8789ea75c671cca (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 01:49:47.096452 18227 tablet_service.cc:1467] Processing CreateTablet for tablet e3116ed5b5254b4d926710cf1e5ed93a (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 01:49:47.102067 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 5090d855c6864a3d904fe1141d503d9d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 01:49:47.109773 18374 tablet_service.cc:1467] Processing CreateTablet for tablet 81a9d50e756d4871a8789ea75c671cca (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 01:49:47.118186 18302 tablet_service.cc:1467] Processing CreateTablet for tablet 8c58e38def504b34b36b61873cc4d5b4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 01:49:47.133028 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5090d855c6864a3d904fe1141d503d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.132643 18301 tablet_service.cc:1467] Processing CreateTablet for tablet d5e09bfc9fdb46af9dd702b4c831e287 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 01:49:47.136642 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 7943deee0b2f49cf8a391e59d9676aa1 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 01:49:47.158507 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 4e46b9ec3add4df0b2026081227bbcd2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 01:49:47.131613 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 4e46b9ec3add4df0b2026081227bbcd2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 01:49:47.160988 18377 tablet_service.cc:1467] Processing CreateTablet for tablet 7943deee0b2f49cf8a391e59d9676aa1 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 01:49:47.161953 18380 tablet_service.cc:1467] Processing CreateTablet for tablet d5e09bfc9fdb46af9dd702b4c831e287 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 01:49:47.163712 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 8c58e38def504b34b36b61873cc4d5b4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 01:49:47.165540 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e46b9ec3add4df0b2026081227bbcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.166172 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7943deee0b2f49cf8a391e59d9676aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.169828 18434 tablet_bootstrap.cc:654] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.170392 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e09bfc9fdb46af9dd702b4c831e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.172709 18375 tablet_service.cc:1467] Processing CreateTablet for tablet 26c1b268530a4a38a3a366888b97b663 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 01:49:47.175508 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c58e38def504b34b36b61873cc4d5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.178325 18228 tablet_service.cc:1467] Processing CreateTablet for tablet 26c1b268530a4a38a3a366888b97b663 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 01:49:47.180485 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a9d50e756d4871a8789ea75c671cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.182744 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e09bfc9fdb46af9dd702b4c831e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.145787 18300 tablet_service.cc:1467] Processing CreateTablet for tablet acc4a55e94ca4287aa230e10d51e21ed (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 01:49:47.146126 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 89b741471cb14483b215e1db8dc72dae (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 01:49:47.183732 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b741471cb14483b215e1db8dc72dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.146345 18224 tablet_service.cc:1467] Processing CreateTablet for tablet cf23c8fedfe4430bbb092f9e927d55e5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 01:49:47.147002 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa5663e2ff0414b948af8d5967266cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.148988 18304 tablet_service.cc:1467] Processing CreateTablet for tablet fd5d4f10e3a24f8e854fda3cbd4b0a0d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 01:49:47.186106 18372 tablet_service.cc:1467] Processing CreateTablet for tablet acc4a55e94ca4287aa230e10d51e21ed (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 01:49:47.187157 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd5d4f10e3a24f8e854fda3cbd4b0a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.187940 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c58e38def504b34b36b61873cc4d5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.189266 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a9d50e756d4871a8789ea75c671cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.190955 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc4a55e94ca4287aa230e10d51e21ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.192169 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3116ed5b5254b4d926710cf1e5ed93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.138458 18303 tablet_service.cc:1467] Processing CreateTablet for tablet 26c1b268530a4a38a3a366888b97b663 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 01:49:47.155774 18219 tablet_service.cc:1467] Processing CreateTablet for tablet d5e09bfc9fdb46af9dd702b4c831e287 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 01:49:47.182412 18306 tablet_service.cc:1467] Processing CreateTablet for tablet a7d24d3149d14b458dd363496c9d7da9 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 01:49:47.193429 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d24d3149d14b458dd363496c9d7da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.194387 18303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c1b268530a4a38a3a366888b97b663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.194849 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e09bfc9fdb46af9dd702b4c831e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.196182 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7943deee0b2f49cf8a391e59d9676aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.196557 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc4a55e94ca4287aa230e10d51e21ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.200165 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e46b9ec3add4df0b2026081227bbcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.208642 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf23c8fedfe4430bbb092f9e927d55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.209039 18381 tablet_service.cc:1467] Processing CreateTablet for tablet fd5d4f10e3a24f8e854fda3cbd4b0a0d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 01:49:47.210314 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd5d4f10e3a24f8e854fda3cbd4b0a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.212138 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c1b268530a4a38a3a366888b97b663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.218950 18435 tablet_bootstrap.cc:654] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.227787 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c1b268530a4a38a3a366888b97b663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.155478 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 8c58e38def504b34b36b61873cc4d5b4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
W20240502 01:49:47.235949 18119 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.165.129:38913, user_credentials={real_user=slave}} blocked reactor thread for 79978.4us
I20240502 01:49:47.236474 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c58e38def504b34b36b61873cc4d5b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:49:47.237229 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:47.239425 18382 tablet_service.cc:1467] Processing CreateTablet for tablet 89b741471cb14483b215e1db8dc72dae (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 01:49:47.241569 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b741471cb14483b215e1db8dc72dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.243505 18378 tablet_service.cc:1467] Processing CreateTablet for tablet a7d24d3149d14b458dd363496c9d7da9 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 01:49:47.244765 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d24d3149d14b458dd363496c9d7da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.246965 18229 tablet_service.cc:1467] Processing CreateTablet for tablet acc4a55e94ca4287aa230e10d51e21ed (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 01:49:47.254613 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 6231687fc05f4510bf8b98b53dca0345 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
W20240502 01:49:47.260028 18118 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 59034.6us
I20240502 01:49:47.260069 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 6231687fc05f4510bf8b98b53dca0345 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 01:49:47.265518 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6231687fc05f4510bf8b98b53dca0345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.265630 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc4a55e94ca4287aa230e10d51e21ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.265758 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 89b741471cb14483b215e1db8dc72dae (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 01:49:47.266984 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b741471cb14483b215e1db8dc72dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.268759 18306 tablet_service.cc:1467] Processing CreateTablet for tablet 97e973320cf04094a84d5530177b70c3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 01:49:47.279808 18227 tablet_service.cc:1467] Processing CreateTablet for tablet fd5d4f10e3a24f8e854fda3cbd4b0a0d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 01:49:47.285292 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6231687fc05f4510bf8b98b53dca0345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.293774 18433 tablet_bootstrap.cc:492] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:47.294319 18433 ts_tablet_manager.cc:1367] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.272s user 0.019s sys 0.008s
I20240502 01:49:47.297482 18433 raft_consensus.cc:348] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:47.298206 18433 raft_consensus.cc:374] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:47.298506 18433 raft_consensus.cc:729] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:47.293998 18304 tablet_service.cc:1467] Processing CreateTablet for tablet ee28c47f80264b0fa5cc15992dfd0f8d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 01:49:47.299875 18228 tablet_service.cc:1467] Processing CreateTablet for tablet a7d24d3149d14b458dd363496c9d7da9 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 01:49:47.300014 18433 consensus_queue.cc:260] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:47.301116 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 42d668f31e144969aae9c1410dd481e6 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 01:49:47.303390 18298 tablet_service.cc:1467] Processing CreateTablet for tablet ba7dd74b7ba34a7dab5ada581af71795 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 01:49:47.305677 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 97e973320cf04094a84d5530177b70c3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 01:49:47.307276 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e973320cf04094a84d5530177b70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.337354 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e973320cf04094a84d5530177b70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.337808 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 6231687fc05f4510bf8b98b53dca0345 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 01:49:47.341552 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7dd74b7ba34a7dab5ada581af71795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.332351 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd5d4f10e3a24f8e854fda3cbd4b0a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.350370 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28c47f80264b0fa5cc15992dfd0f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.355068 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d24d3149d14b458dd363496c9d7da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.357964 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d668f31e144969aae9c1410dd481e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:49:47.363852 18118 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 61053.2us
W20240502 01:49:47.364058 18119 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 60256.9us
I20240502 01:49:47.371706 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6231687fc05f4510bf8b98b53dca0345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.382174 18433 ts_tablet_manager.cc:1398] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.088s user 0.013s sys 0.000s
I20240502 01:49:47.382997 18418 heartbeater.cc:503] Master 127.16.165.190:35883 was elected leader, sending a full tablet report...
I20240502 01:49:47.383293 18433 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:47.389716 18433 tablet_bootstrap.cc:654] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.399336 18377 tablet_service.cc:1467] Processing CreateTablet for tablet ba7dd74b7ba34a7dab5ada581af71795 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 01:49:47.400652 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7dd74b7ba34a7dab5ada581af71795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.404662 18374 tablet_service.cc:1467] Processing CreateTablet for tablet ee28c47f80264b0fa5cc15992dfd0f8d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 01:49:47.405931 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28c47f80264b0fa5cc15992dfd0f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.420620 18220 tablet_service.cc:1467] Processing CreateTablet for tablet ee28c47f80264b0fa5cc15992dfd0f8d (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 01:49:47.421957 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28c47f80264b0fa5cc15992dfd0f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.426038 18265 heartbeater.cc:503] Master 127.16.165.190:35883 was elected leader, sending a full tablet report...
I20240502 01:49:47.427150 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 42d668f31e144969aae9c1410dd481e6 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 01:49:47.432459 18225 tablet_service.cc:1467] Processing CreateTablet for tablet ba7dd74b7ba34a7dab5ada581af71795 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 01:49:47.433734 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7dd74b7ba34a7dab5ada581af71795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.446128 18380 tablet_service.cc:1467] Processing CreateTablet for tablet bac68a1674ab4b15a8274884f20a0266 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 01:49:47.427196 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 42d668f31e144969aae9c1410dd481e6 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 01:49:47.453230 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d668f31e144969aae9c1410dd481e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.454269 18222 tablet_service.cc:1467] Processing CreateTablet for tablet bac68a1674ab4b15a8274884f20a0266 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 01:49:47.455451 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac68a1674ab4b15a8274884f20a0266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.457803 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 0d731e8bc3f84af8aef9c511ae2e78ca (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 01:49:47.459282 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d731e8bc3f84af8aef9c511ae2e78ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.473996 18375 tablet_service.cc:1467] Processing CreateTablet for tablet 0d731e8bc3f84af8aef9c511ae2e78ca (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 01:49:47.475366 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d731e8bc3f84af8aef9c511ae2e78ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.476416 18434 tablet_bootstrap.cc:492] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:47.476938 18434 ts_tablet_manager.cc:1367] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.402s user 0.011s sys 0.025s
I20240502 01:49:47.477919 18374 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcca9e765354912af2383d0f8f8540a (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 01:49:47.479197 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcca9e765354912af2383d0f8f8540a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.480356 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac68a1674ab4b15a8274884f20a0266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.485334 18382 tablet_service.cc:1467] Processing CreateTablet for tablet a92fc4beef0b4747bcf45cbabe4bb118 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 01:49:47.486580 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92fc4beef0b4747bcf45cbabe4bb118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.464239 18226 tablet_service.cc:1467] Processing CreateTablet for tablet 97e973320cf04094a84d5530177b70c3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 01:49:47.488888 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e973320cf04094a84d5530177b70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.479579 18434 raft_consensus.cc:348] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.499002 18434 raft_consensus.cc:374] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:47.499352 18434 raft_consensus.cc:729] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:47.504884 18381 tablet_service.cc:1467] Processing CreateTablet for tablet 8936a88f18dd4d1185402793335458f2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 01:49:47.506150 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d668f31e144969aae9c1410dd481e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.507973 18224 tablet_service.cc:1467] Processing CreateTablet for tablet a92fc4beef0b4747bcf45cbabe4bb118 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 01:49:47.509299 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92fc4beef0b4747bcf45cbabe4bb118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.510825 18434 consensus_queue.cc:260] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.515599 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcca9e765354912af2383d0f8f8540a (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 01:49:47.517371 18227 tablet_service.cc:1467] Processing CreateTablet for tablet 282fd6d7ad0249509f67bfcbc4c0adf5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 01:49:47.517926 18342 heartbeater.cc:503] Master 127.16.165.190:35883 was elected leader, sending a full tablet report...
I20240502 01:49:47.536073 18438 raft_consensus.cc:482] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:47.536537 18438 raft_consensus.cc:504] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:47.538643 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcca9e765354912af2383d0f8f8540a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.539263 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282fd6d7ad0249509f67bfcbc4c0adf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.553987 18435 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:47.554597 18435 ts_tablet_manager.cc:1367] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.486s user 0.013s sys 0.011s
I20240502 01:49:47.522115 18219 tablet_service.cc:1467] Processing CreateTablet for tablet ae19377bfe6e48e481f6054c06e669b6 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20240502 01:49:47.530490 18119 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.165.129:38913, user_credentials={real_user=slave}} blocked reactor thread for 63543.4us
I20240502 01:49:47.514853 18305 tablet_service.cc:1467] Processing CreateTablet for tablet bac68a1674ab4b15a8274884f20a0266 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 01:49:47.568439 18306 tablet_service.cc:1467] Processing CreateTablet for tablet 0d731e8bc3f84af8aef9c511ae2e78ca (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 01:49:47.566948 18438 leader_election.cc:288] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:49:47.585847 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19377bfe6e48e481f6054c06e669b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.590835 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d731e8bc3f84af8aef9c511ae2e78ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.592103 18377 tablet_service.cc:1467] Processing CreateTablet for tablet 45daae4b488442c797a69bb638169fc4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 01:49:47.594432 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 45daae4b488442c797a69bb638169fc4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 01:49:47.596447 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 73530ff3121b43478be31af22ba77d09 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 01:49:47.598273 18372 tablet_service.cc:1467] Processing CreateTablet for tablet ae19377bfe6e48e481f6054c06e669b6 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 01:49:47.600852 18433 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:47.601329 18433 ts_tablet_manager.cc:1367] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.218s user 0.019s sys 0.022s
I20240502 01:49:47.610471 18442 raft_consensus.cc:482] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:47.600615 18382 tablet_service.cc:1467] Processing CreateTablet for tablet 420ebc81e61440d997f99ad838b88f79 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 01:49:47.610991 18442 raft_consensus.cc:504] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.601773 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8936a88f18dd4d1185402793335458f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.626264 18442 leader_election.cc:288] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:47.626593 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45daae4b488442c797a69bb638169fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.631141 18374 tablet_service.cc:1467] Processing CreateTablet for tablet c3b45ab88cf544fca9ca712cfaa2c601 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 01:49:47.617993 18304 tablet_service.cc:1467] Processing CreateTablet for tablet a92fc4beef0b4747bcf45cbabe4bb118 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 01:49:47.652405 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92fc4beef0b4747bcf45cbabe4bb118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.652602 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 282fd6d7ad0249509f67bfcbc4c0adf5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 01:49:47.653419 18433 raft_consensus.cc:348] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.654177 18433 raft_consensus.cc:374] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:47.654474 18433 raft_consensus.cc:729] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:47.655177 18433 consensus_queue.cc:260] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.674765 18433 ts_tablet_manager.cc:1398] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.060s user 0.002s sys 0.003s
I20240502 01:49:47.656736 18228 tablet_service.cc:1467] Processing CreateTablet for tablet 73530ff3121b43478be31af22ba77d09 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 01:49:47.676617 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73530ff3121b43478be31af22ba77d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.658679 18225 tablet_service.cc:1467] Processing CreateTablet for tablet c3b45ab88cf544fca9ca712cfaa2c601 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 01:49:47.680799 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b45ab88cf544fca9ca712cfaa2c601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.661957 18435 raft_consensus.cc:348] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.670640 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45daae4b488442c797a69bb638169fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.684414 18435 raft_consensus.cc:374] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:47.684739 18435 raft_consensus.cc:729] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:47.675702 18433 tablet_bootstrap.cc:492] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:47.684242 18226 tablet_service.cc:1467] Processing CreateTablet for tablet 8d66ef8312fd46c79932b145f2b6197b (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 01:49:47.692361 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d66ef8312fd46c79932b145f2b6197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.693289 18303 tablet_service.cc:1467] Processing CreateTablet for tablet 282fd6d7ad0249509f67bfcbc4c0adf5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 01:49:47.693616 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 8d66ef8312fd46c79932b145f2b6197b (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 01:49:47.694532 18303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282fd6d7ad0249509f67bfcbc4c0adf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.695353 18435 consensus_queue.cc:260] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.699167 18300 tablet_service.cc:1467] Processing CreateTablet for tablet ae19377bfe6e48e481f6054c06e669b6 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 01:49:47.700430 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19377bfe6e48e481f6054c06e669b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.691664 18301 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcca9e765354912af2383d0f8f8540a (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 01:49:47.703289 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcca9e765354912af2383d0f8f8540a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.705621 18441 raft_consensus.cc:482] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:47.706099 18441 raft_consensus.cc:504] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.707255 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 0e766a1489b54d248cac1c3254afcdb3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 01:49:47.684942 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19377bfe6e48e481f6054c06e669b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.710556 18302 tablet_service.cc:1467] Processing CreateTablet for tablet 45daae4b488442c797a69bb638169fc4 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 01:49:47.720199 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b45ab88cf544fca9ca712cfaa2c601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.712352 18441 leader_election.cc:288] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:47.714874 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac68a1674ab4b15a8274884f20a0266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.720587 18434 ts_tablet_manager.cc:1398] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.243s user 0.024s sys 0.019s
I20240502 01:49:47.721712 18434 tablet_bootstrap.cc:492] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:47.724246 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45daae4b488442c797a69bb638169fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.724828 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420ebc81e61440d997f99ad838b88f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.727088 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282fd6d7ad0249509f67bfcbc4c0adf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.732836 18299 tablet_service.cc:1467] Processing CreateTablet for tablet c3b45ab88cf544fca9ca712cfaa2c601 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 01:49:47.734090 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b45ab88cf544fca9ca712cfaa2c601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.748531 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e766a1489b54d248cac1c3254afcdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.752194 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73530ff3121b43478be31af22ba77d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.755427 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a75f4d1c0c746f3ba51960f4f4e4f55" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:47.758024 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba72fe24abe4fe893e60d48e6679580" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:47.758811 18238 raft_consensus.cc:2454] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
W20240502 01:49:47.759963 18356 leader_election.cc:341] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:47.761684 18356 leader_election.cc:302] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:47.762483 18441 raft_consensus.cc:2781] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:47.762831 18441 raft_consensus.cc:482] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:47.763167 18441 raft_consensus.cc:3037] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:47.778630 18300 tablet_service.cc:1467] Processing CreateTablet for tablet 73530ff3121b43478be31af22ba77d09 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 01:49:47.781239 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73530ff3121b43478be31af22ba77d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.768955 18433 tablet_bootstrap.cc:654] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.797674 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 8936a88f18dd4d1185402793335458f2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 01:49:47.798988 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8936a88f18dd4d1185402793335458f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.684238 18227 tablet_service.cc:1467] Processing CreateTablet for tablet 420ebc81e61440d997f99ad838b88f79 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 01:49:47.800917 18435 ts_tablet_manager.cc:1398] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.245s user 0.008s sys 0.000s
I20240502 01:49:47.769681 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d66ef8312fd46c79932b145f2b6197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.804682 18435 tablet_bootstrap.cc:492] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:47.807060 18375 tablet_service.cc:1467] Processing CreateTablet for tablet 92c0d6ed72f045a3b39631b83a054d78 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 01:49:47.810770 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 92c0d6ed72f045a3b39631b83a054d78 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 01:49:47.816771 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a75f4d1c0c746f3ba51960f4f4e4f55" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:47.817723 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba72fe24abe4fe893e60d48e6679580" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:47.825644 18304 tablet_service.cc:1467] Processing CreateTablet for tablet 8d66ef8312fd46c79932b145f2b6197b (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
W20240502 01:49:47.826573 18355 leader_election.cc:341] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:47.826975 18434 tablet_bootstrap.cc:654] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.798585 18374 tablet_service.cc:1467] Processing CreateTablet for tablet b5411115a92043f684df0c01d3cc5e36 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 01:49:47.685523 18222 tablet_service.cc:1467] Processing CreateTablet for tablet 8936a88f18dd4d1185402793335458f2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 01:49:47.826938 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d66ef8312fd46c79932b145f2b6197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.836328 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 810e60f2654045a39491fd7e358029da (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 01:49:47.836633 18298 tablet_service.cc:1467] Processing CreateTablet for tablet 420ebc81e61440d997f99ad838b88f79 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 01:49:47.837870 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420ebc81e61440d997f99ad838b88f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.840644 18220 tablet_service.cc:1467] Processing CreateTablet for tablet b5411115a92043f684df0c01d3cc5e36 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 01:49:47.841742 18302 tablet_service.cc:1467] Processing CreateTablet for tablet 0e766a1489b54d248cac1c3254afcdb3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 01:49:47.843011 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e766a1489b54d248cac1c3254afcdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.851244 18355 leader_election.cc:302] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:47.852187 18438 raft_consensus.cc:2726] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:49:47.853367 18355 leader_election.cc:341] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:47.856235 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420ebc81e61440d997f99ad838b88f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.858558 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e60f2654045a39491fd7e358029da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.860282 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8936a88f18dd4d1185402793335458f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.862607 18306 tablet_service.cc:1467] Processing CreateTablet for tablet b5411115a92043f684df0c01d3cc5e36 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 01:49:47.864529 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c0d6ed72f045a3b39631b83a054d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.865310 18441 raft_consensus.cc:504] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.866078 18435 tablet_bootstrap.cc:654] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:47.868229 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5411115a92043f684df0c01d3cc5e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.871490 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5411115a92043f684df0c01d3cc5e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.875831 18382 tablet_service.cc:1467] Processing CreateTablet for tablet 7f441096d7e74258a6c1629ba45bb1f3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 01:49:47.876987 18301 tablet_service.cc:1467] Processing CreateTablet for tablet 92c0d6ed72f045a3b39631b83a054d78 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 01:49:47.883443 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c0d6ed72f045a3b39631b83a054d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.886639 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 9be322d2860f40408a8cd8199731477c (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 01:49:47.887619 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 810e60f2654045a39491fd7e358029da (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 01:49:47.884825 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 0e766a1489b54d248cac1c3254afcdb3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 01:49:47.890406 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f441096d7e74258a6c1629ba45bb1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.889065 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 810e60f2654045a39491fd7e358029da (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 01:49:47.898213 18225 tablet_service.cc:1467] Processing CreateTablet for tablet cdd8414d528f4407b7369da924f707c5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 01:49:47.898561 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:47.899437 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd8414d528f4407b7369da924f707c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:49:47.900957 18280 leader_election.cc:341] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:47.901422 18228 tablet_service.cc:1467] Processing CreateTablet for tablet a3a4278616b44d738b2b7aa7eafef008 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 01:49:47.902581 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a4278616b44d738b2b7aa7eafef008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.900876 18304 tablet_service.cc:1467] Processing CreateTablet for tablet 9be322d2860f40408a8cd8199731477c (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 01:49:47.904812 18441 leader_election.cc:288] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:47.905886 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba72fe24abe4fe893e60d48e6679580" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:47.906037 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 9be322d2860f40408a8cd8199731477c (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 01:49:47.906509 18238 raft_consensus.cc:3037] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:47.912240 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:47.912976 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be322d2860f40408a8cd8199731477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.906859 18302 tablet_service.cc:1467] Processing CreateTablet for tablet cdd8414d528f4407b7369da924f707c5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20240502 01:49:47.916643 18280 leader_election.cc:341] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:47.917188 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba72fe24abe4fe893e60d48e6679580" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:47.917265 18280 leader_election.cc:302] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:47.918529 18442 raft_consensus.cc:2726] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:49:47.918596 18355 leader_election.cc:341] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:47.925105 18224 tablet_service.cc:1467] Processing CreateTablet for tablet 22b1b170e56d4e1591f3991e0883ce16 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 01:49:47.926378 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b1b170e56d4e1591f3991e0883ce16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.931391 18238 raft_consensus.cc:2454] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:47.933472 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5411115a92043f684df0c01d3cc5e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.935249 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be322d2860f40408a8cd8199731477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.936132 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e766a1489b54d248cac1c3254afcdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.937763 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e60f2654045a39491fd7e358029da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.940523 18356 leader_election.cc:302] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:47.942258 18441 raft_consensus.cc:2781] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:47.942715 18441 raft_consensus.cc:686] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:47.944725 18441 consensus_queue.cc:237] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.947433 18372 tablet_service.cc:1467] Processing CreateTablet for tablet cdd8414d528f4407b7369da924f707c5 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 01:49:47.953104 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd8414d528f4407b7369da924f707c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.961886 18433 tablet_bootstrap.cc:492] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:47.962375 18433 ts_tablet_manager.cc:1367] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.287s user 0.008s sys 0.015s
I20240502 01:49:47.956213 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c0d6ed72f045a3b39631b83a054d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.966420 18433 raft_consensus.cc:348] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.967844 18433 raft_consensus.cc:374] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:47.968238 18433 raft_consensus.cc:729] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:47.966421 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 7f441096d7e74258a6c1629ba45bb1f3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 01:49:47.969588 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f441096d7e74258a6c1629ba45bb1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.980110 18434 tablet_bootstrap.cc:492] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:47.980619 18434 ts_tablet_manager.cc:1367] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.259s user 0.023s sys 0.021s
I20240502 01:49:47.983176 18434 raft_consensus.cc:348] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.983882 18434 raft_consensus.cc:374] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:47.984181 18434 raft_consensus.cc:729] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:47.985162 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 22b1b170e56d4e1591f3991e0883ce16 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 01:49:47.985303 18434 consensus_queue.cc:260] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.986526 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b1b170e56d4e1591f3991e0883ce16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.990226 18434 ts_tablet_manager.cc:1398] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:49:47.991140 18434 tablet_bootstrap.cc:492] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:47.992044 18433 consensus_queue.cc:260] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.994406 18433 ts_tablet_manager.cc:1398] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20240502 01:49:47.994558 18438 raft_consensus.cc:482] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:47.995039 18438 raft_consensus.cc:504] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:47.997285 18382 tablet_service.cc:1467] Processing CreateTablet for tablet a3a4278616b44d738b2b7aa7eafef008 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 01:49:47.998502 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a4278616b44d738b2b7aa7eafef008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:47.998634 18438 leader_election.cc:288] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.000934 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54a35564ce4389a2395ea0832a968b" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.002142 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54a35564ce4389a2395ea0832a968b" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.002763 18315 raft_consensus.cc:2454] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:48.003934 18355 leader_election.cc:302] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
W20240502 01:49:48.005317 18356 leader_election.cc:341] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.007064 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 084a332369434663a3a306d0fb57e927 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 01:49:48.008685 18438 raft_consensus.cc:2781] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.009060 18438 raft_consensus.cc:482] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.009377 18438 raft_consensus.cc:3037] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.010591 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be322d2860f40408a8cd8199731477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.011206 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd8414d528f4407b7369da924f707c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.014767 18433 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.016479 18226 tablet_service.cc:1467] Processing CreateTablet for tablet 084a332369434663a3a306d0fb57e927 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 01:49:48.017706 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084a332369434663a3a306d0fb57e927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.018687 18438 raft_consensus.cc:504] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.020182 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e60f2654045a39491fd7e358029da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.020612 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084a332369434663a3a306d0fb57e927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.023255 18433 tablet_bootstrap.cc:654] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.077807 18298 tablet_service.cc:1467] Processing CreateTablet for tablet 7f441096d7e74258a6c1629ba45bb1f3 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 01:49:48.078835 18381 tablet_service.cc:1467] Processing CreateTablet for tablet cf2837217d424de797067c2daede656c (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 01:49:48.070027 18300 tablet_service.cc:1467] Processing CreateTablet for tablet a3a4278616b44d738b2b7aa7eafef008 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 01:49:48.089442 18130 catalog_manager.cc:5484] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:48.093971 18377 tablet_service.cc:1467] Processing CreateTablet for tablet 09b5bb05d88c44618ab1fde55d4dd7a2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 01:49:48.096091 18438 leader_election.cc:288] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.097062 18434 tablet_bootstrap.cc:654] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.097196 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54a35564ce4389a2395ea0832a968b" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:48.097803 18315 raft_consensus.cc:3037] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.098186 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54a35564ce4389a2395ea0832a968b" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
W20240502 01:49:48.099464 18356 leader_election.cc:341] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.099675 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a4278616b44d738b2b7aa7eafef008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.095623 18229 tablet_service.cc:1467] Processing CreateTablet for tablet cf2837217d424de797067c2daede656c (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 01:49:48.102852 18303 tablet_service.cc:1467] Processing CreateTablet for tablet 22b1b170e56d4e1591f3991e0883ce16 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 01:49:48.104756 18303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b1b170e56d4e1591f3991e0883ce16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.105264 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2837217d424de797067c2daede656c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.107550 18315 raft_consensus.cc:2454] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:48.122193 18355 leader_election.cc:302] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:48.127414 18438 raft_consensus.cc:2781] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:48.127949 18438 raft_consensus.cc:686] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:48.128046 18306 tablet_service.cc:1467] Processing CreateTablet for tablet 084a332369434663a3a306d0fb57e927 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 01:49:48.128801 18438 consensus_queue.cc:237] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.135090 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2837217d424de797067c2daede656c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.140578 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b5bb05d88c44618ab1fde55d4dd7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.152148 18222 tablet_service.cc:1467] Processing CreateTablet for tablet 09b5bb05d88c44618ab1fde55d4dd7a2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 01:49:48.152686 18297 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54726 (request call id 40) took 1028 ms. Trace:
I20240502 01:49:48.136348 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f441096d7e74258a6c1629ba45bb1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.158386 18433 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.170683 18433 ts_tablet_manager.cc:1367] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.018s
I20240502 01:49:48.168519 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 2318233df2844c48918882f56859c152 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 01:49:48.168779 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 7210798da7624de29528dfd61b8a1efa (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 01:49:48.175207 18435 tablet_bootstrap.cc:492] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.175707 18435 ts_tablet_manager.cc:1367] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.371s user 0.010s sys 0.019s
I20240502 01:49:48.177750 18433 raft_consensus.cc:348] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.178284 18435 raft_consensus.cc:348] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.178803 18433 raft_consensus.cc:374] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.179141 18433 raft_consensus.cc:729] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.172075 18297 rpcz_store.cc:276] 0502 01:49:47.124363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:49:47.796829 (+672466us) service_pool.cc:225] Handling call
0502 01:49:47.798563 (+ 1734us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 01:49:47.798662 (+ 99us) ts_tablet_manager.cc:655] Creating new metadata...
0502 01:49:48.005805 (+207143us) tablet_metadata.cc:641] Metadata flushed
0502 01:49:48.152167 (+146362us) tablet_replica.cc:205] Creating consensus instance
0502 01:49:48.152623 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 01:49:48.181368 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7210798da7624de29528dfd61b8a1efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.181181 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 09b5bb05d88c44618ab1fde55d4dd7a2 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 01:49:48.183022 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2318233df2844c48918882f56859c152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.184053 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 7210798da7624de29528dfd61b8a1efa (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 01:49:48.185093 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084a332369434663a3a306d0fb57e927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.185423 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7210798da7624de29528dfd61b8a1efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.187012 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b5bb05d88c44618ab1fde55d4dd7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.191740 18224 tablet_service.cc:1467] Processing CreateTablet for tablet 6049ac47c1fb40289b45cadccb8ecf97 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 01:49:48.192410 18227 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41696 (request call id 38) took 1038 ms. Trace:
I20240502 01:49:48.193321 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6049ac47c1fb40289b45cadccb8ecf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.180095 18433 consensus_queue.cc:260] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.194036 18302 tablet_service.cc:1467] Processing CreateTablet for tablet 7210798da7624de29528dfd61b8a1efa (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 01:49:48.196888 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9cb5b8b9404935a860997c60c29470 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 01:49:48.179060 18435 raft_consensus.cc:374] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.197507 18299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54726 (request call id 39) took 1080 ms. Trace:
I20240502 01:49:48.143579 18301 tablet_service.cc:1467] Processing CreateTablet for tablet cf2837217d424de797067c2daede656c (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 01:49:48.199566 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9cb5b8b9404935a860997c60c29470 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 01:49:48.192811 18227 rpcz_store.cc:276] 0502 01:49:47.154315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:49:47.651719 (+497404us) service_pool.cc:225] Handling call
0502 01:49:47.799722 (+148003us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 01:49:47.799832 (+ 110us) ts_tablet_manager.cc:655] Creating new metadata...
0502 01:49:48.097332 (+297500us) tablet_metadata.cc:641] Metadata flushed
0502 01:49:48.191902 (+ 94570us) tablet_replica.cc:205] Creating consensus instance
0502 01:49:48.192356 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 01:49:48.208644 18304 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9cb5b8b9404935a860997c60c29470 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 01:49:48.210635 18228 tablet_service.cc:1467] Processing CreateTablet for tablet 2318233df2844c48918882f56859c152 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 01:49:48.211934 18435 raft_consensus.cc:729] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.217314 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b5bb05d88c44618ab1fde55d4dd7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.223191 18433 ts_tablet_manager.cc:1398] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.052s user 0.005s sys 0.003s
I20240502 01:49:48.224757 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9cb5b8b9404935a860997c60c29470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.212713 18299 rpcz_store.cc:276] 0502 01:49:47.117271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:49:47.731839 (+614568us) service_pool.cc:225] Handling call
0502 01:49:47.733635 (+ 1796us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 01:49:47.733732 (+ 97us) ts_tablet_manager.cc:655] Creating new metadata...
0502 01:49:48.022885 (+289153us) tablet_metadata.cc:641] Metadata flushed
0502 01:49:48.196315 (+173430us) tablet_replica.cc:205] Creating consensus instance
0502 01:49:48.197446 (+ 1131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 01:49:48.227051 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 6049ac47c1fb40289b45cadccb8ecf97 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 01:49:48.228159 18435 consensus_queue.cc:260] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.234364 18435 ts_tablet_manager.cc:1398] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.058s user 0.004s sys 0.002s
I20240502 01:49:48.235328 18435 tablet_bootstrap.cc:492] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.217593 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2837217d424de797067c2daede656c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.238786 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6049ac47c1fb40289b45cadccb8ecf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.220347 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9cb5b8b9404935a860997c60c29470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.220986 18434 tablet_bootstrap.cc:492] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.221211 18300 tablet_service.cc:1467] Processing CreateTablet for tablet 2318233df2844c48918882f56859c152 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 01:49:48.221863 18374 tablet_service.cc:1467] Processing CreateTablet for tablet 6049ac47c1fb40289b45cadccb8ecf97 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 01:49:48.242942 18433 tablet_bootstrap.cc:492] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.248679 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 4c634b7903104fb1aebaa8b7722fd417 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20240502 01:49:48.250799 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:48.252162 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7210798da7624de29528dfd61b8a1efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.252801 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c634b7903104fb1aebaa8b7722fd417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.254738 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6049ac47c1fb40289b45cadccb8ecf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.254859 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2318233df2844c48918882f56859c152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.197602 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 4c634b7903104fb1aebaa8b7722fd417 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 01:49:48.263943 18434 ts_tablet_manager.cc:1367] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.273s user 0.012s sys 0.001s
I20240502 01:49:48.270862 18433 tablet_bootstrap.cc:654] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.279320 18435 tablet_bootstrap.cc:654] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.211427 18373 tablet_service.cc:1467] Processing CreateTablet for tablet 4c634b7903104fb1aebaa8b7722fd417 (DEFAULT_TABLE table=test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 01:49:48.280845 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c634b7903104fb1aebaa8b7722fd417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.281859 18434 raft_consensus.cc:348] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.282618 18434 raft_consensus.cc:374] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.282915 18434 raft_consensus.cc:729] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.288357 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2318233df2844c48918882f56859c152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.288491 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9cb5b8b9404935a860997c60c29470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.297849 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c634b7903104fb1aebaa8b7722fd417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:48.287945 18434 consensus_queue.cc:260] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.315462 18434 ts_tablet_manager.cc:1398] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.051s user 0.004s sys 0.003s
I20240502 01:49:48.327708 18434 tablet_bootstrap.cc:492] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.343453 18434 tablet_bootstrap.cc:654] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.355491 18433 tablet_bootstrap.cc:492] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.356060 18433 ts_tablet_manager.cc:1367] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.132s user 0.032s sys 0.015s
I20240502 01:49:48.358557 18433 raft_consensus.cc:348] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.359120 18433 raft_consensus.cc:374] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.359406 18433 raft_consensus.cc:729] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.360163 18433 consensus_queue.cc:260] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.362700 18433 ts_tablet_manager.cc:1398] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:49:48.363796 18433 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.370780 18435 tablet_bootstrap.cc:492] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.371057 18433 tablet_bootstrap.cc:654] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.371287 18435 ts_tablet_manager.cc:1367] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.136s user 0.020s sys 0.000s
I20240502 01:49:48.374995 18435 raft_consensus.cc:348] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.375733 18435 raft_consensus.cc:374] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.376060 18435 raft_consensus.cc:729] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.377559 18435 consensus_queue.cc:260] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.379482 18435 ts_tablet_manager.cc:1398] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:49:48.380609 18435 tablet_bootstrap.cc:492] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.383003 18434 tablet_bootstrap.cc:492] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.383524 18434 ts_tablet_manager.cc:1367] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.009s
I20240502 01:49:48.386581 18435 tablet_bootstrap.cc:654] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.386232 18434 raft_consensus.cc:348] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.387061 18434 raft_consensus.cc:374] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.387382 18434 raft_consensus.cc:729] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.388466 18433 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.388214 18434 consensus_queue.cc:260] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.388947 18433 ts_tablet_manager.cc:1367] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.006s
I20240502 01:49:48.389842 18133 catalog_manager.cc:5484] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:48.391690 18433 raft_consensus.cc:348] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.394294 18434 ts_tablet_manager.cc:1398] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:49:48.396376 18433 raft_consensus.cc:374] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.396561 18434 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.396732 18433 raft_consensus.cc:729] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.397914 18435 tablet_bootstrap.cc:492] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.397501 18433 consensus_queue.cc:260] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.398386 18435 ts_tablet_manager.cc:1367] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240502 01:49:48.401150 18433 ts_tablet_manager.cc:1398] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240502 01:49:48.402172 18433 tablet_bootstrap.cc:492] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.402848 18434 tablet_bootstrap.cc:654] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.403203 18442 raft_consensus.cc:482] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.403013 18435 raft_consensus.cc:348] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.403806 18435 raft_consensus.cc:374] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.404124 18435 raft_consensus.cc:729] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.403676 18442 raft_consensus.cc:504] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.404954 18435 consensus_queue.cc:260] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.406453 18442 leader_election.cc:288] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.409516 18433 tablet_bootstrap.cc:654] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.409353 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a06b4fbd949e490a4e206fa313777" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.410192 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a06b4fbd949e490a4e206fa313777" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.410827 18392 raft_consensus.cc:2454] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
W20240502 01:49:48.413195 18280 leader_election.cc:341] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.414652 18280 leader_election.cc:302] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:48.417528 18451 raft_consensus.cc:482] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.418031 18451 raft_consensus.cc:504] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.426141 18442 raft_consensus.cc:2781] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.426681 18442 raft_consensus.cc:482] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.427078 18442 raft_consensus.cc:3037] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.432183 18442 raft_consensus.cc:504] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:48.432721 18355 consensus_peers.cc:485] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:48.434831 18451 leader_election.cc:288] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.435672 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a06b4fbd949e490a4e206fa313777" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:48.436380 18392 raft_consensus.cc:3037] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.443353 18451 raft_consensus.cc:482] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.444896 18392 raft_consensus.cc:2454] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:48.445173 18451 raft_consensus.cc:504] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.446131 18280 leader_election.cc:302] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:48.447576 18451 leader_election.cc:288] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.449563 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a06b4fbd949e490a4e206fa313777" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:48.449939 18442 leader_election.cc:288] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.450302 18475 raft_consensus.cc:2781] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:49:48.450846 18280 leader_election.cc:341] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.455561 18475 raft_consensus.cc:686] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:48.456987 18475 consensus_queue.cc:237] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.453572 18435 ts_tablet_manager.cc:1398] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.055s user 0.006s sys 0.001s
I20240502 01:49:48.459941 18435 tablet_bootstrap.cc:492] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.462633 18441 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:48.464012 18438 raft_consensus.cc:482] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.464488 18438 raft_consensus.cc:504] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.467653 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.469049 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54a35564ce4389a2395ea0832a968b" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.470139 18438 leader_election.cc:288] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:48.471200 18355 leader_election.cc:341] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.471827 18435 tablet_bootstrap.cc:654] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.472877 18434 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.473486 18434 ts_tablet_manager.cc:1367] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.032s
I20240502 01:49:48.473588 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.472837 18433 tablet_bootstrap.cc:492] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
W20240502 01:49:48.475205 18356 leader_election.cc:341] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.475780 18433 ts_tablet_manager.cc:1367] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.074s user 0.029s sys 0.039s
I20240502 01:49:48.475689 18356 leader_election.cc:302] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:48.476500 18434 raft_consensus.cc:348] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.477491 18434 raft_consensus.cc:374] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.476879 18441 raft_consensus.cc:2726] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.477823 18434 raft_consensus.cc:729] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.481848 18434 consensus_queue.cc:260] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.484087 18434 ts_tablet_manager.cc:1398] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240502 01:49:48.485009 18434 tablet_bootstrap.cc:492] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.485800 18132 catalog_manager.cc:5484] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:48.487326 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7889fa2a16487d8b767d51bb21abf2" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:48.489482 18203 leader_election.cc:341] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.485795 18433 raft_consensus.cc:348] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.490245 18475 raft_consensus.cc:482] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.490836 18475 raft_consensus.cc:504] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.492403 18434 tablet_bootstrap.cc:654] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.492903 18475 leader_election.cc:288] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.495664 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54a35564ce4389a2395ea0832a968b" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.496470 18316 raft_consensus.cc:2379] T da54a35564ce4389a2395ea0832a968b P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:48.496259 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7889fa2a16487d8b767d51bb21abf2" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.490657 18433 raft_consensus.cc:374] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.497794 18433 raft_consensus.cc:729] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.498363 18201 leader_election.cc:302] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.498538 18433 consensus_queue.cc:260] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:48.499583 18201 leader_election.cc:341] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.500025 18201 leader_election.cc:302] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.500177 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:48.501345 18343 tablet.cc:2369] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:49:48.501605 18451 raft_consensus.cc:3037] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.504137 18433 ts_tablet_manager.cc:1398] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.028s user 0.004s sys 0.003s
W20240502 01:49:48.505124 18280 leader_election.cc:341] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.505429 18433 tablet_bootstrap.cc:492] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.508157 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:48.509577 18280 leader_election.cc:341] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.510035 18280 leader_election.cc:302] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.511675 18433 tablet_bootstrap.cc:654] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.514752 18475 raft_consensus.cc:2726] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.516757 18435 tablet_bootstrap.cc:492] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.517138 18435 ts_tablet_manager.cc:1367] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.002s
I20240502 01:49:48.519052 18435 raft_consensus.cc:348] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.519564 18435 raft_consensus.cc:374] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.519770 18435 raft_consensus.cc:729] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.523679 18451 raft_consensus.cc:2726] T da54a35564ce4389a2395ea0832a968b P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.524998 18451 raft_consensus.cc:2726] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.525648 18435 consensus_queue.cc:260] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.529824 18435 ts_tablet_manager.cc:1398] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20240502 01:49:48.530776 18435 tablet_bootstrap.cc:492] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.539520 18434 tablet_bootstrap.cc:492] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.540084 18434 ts_tablet_manager.cc:1367] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.055s user 0.026s sys 0.011s
I20240502 01:49:48.547991 18435 tablet_bootstrap.cc:654] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.548400 18434 raft_consensus.cc:348] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.549168 18434 raft_consensus.cc:374] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.549469 18434 raft_consensus.cc:729] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.550277 18433 tablet_bootstrap.cc:492] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.550231 18434 consensus_queue.cc:260] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.550956 18433 ts_tablet_manager.cc:1367] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.009s
I20240502 01:49:48.554111 18433 raft_consensus.cc:348] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.555198 18433 raft_consensus.cc:374] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.555657 18434 ts_tablet_manager.cc:1398] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240502 01:49:48.555894 18433 raft_consensus.cc:729] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.556731 18434 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.557498 18433 consensus_queue.cc:260] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.560649 18433 ts_tablet_manager.cc:1398] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:49:48.562953 18434 tablet_bootstrap.cc:654] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.563980 18433 tablet_bootstrap.cc:492] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.573940 18433 tablet_bootstrap.cc:654] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.586760 18433 tablet_bootstrap.cc:492] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.586771 18435 tablet_bootstrap.cc:492] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.587193 18434 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.587303 18433 ts_tablet_manager.cc:1367] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.000s
I20240502 01:49:48.587304 18435 ts_tablet_manager.cc:1367] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.026s
I20240502 01:49:48.587626 18434 ts_tablet_manager.cc:1367] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.012s
I20240502 01:49:48.589749 18434 raft_consensus.cc:348] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.590309 18434 raft_consensus.cc:374] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.590623 18434 raft_consensus.cc:729] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.590087 18435 raft_consensus.cc:348] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.590184 18433 raft_consensus.cc:348] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.590967 18435 raft_consensus.cc:374] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.591018 18433 raft_consensus.cc:374] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.591396 18435 raft_consensus.cc:729] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.591459 18433 raft_consensus.cc:729] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.591223 18434 consensus_queue.cc:260] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.592235 18441 raft_consensus.cc:482] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.592209 18435 consensus_queue.cc:260] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.592782 18441 raft_consensus.cc:504] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.593673 18434 ts_tablet_manager.cc:1398] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:49:48.594516 18435 ts_tablet_manager.cc:1398] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240502 01:49:48.594735 18441 leader_election.cc:288] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.594964 18434 tablet_bootstrap.cc:492] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.595404 18435 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.597090 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.598229 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.598850 18315 raft_consensus.cc:2454] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:48.601058 18434 tablet_bootstrap.cc:654] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.601202 18435 tablet_bootstrap.cc:654] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:48.603361 18356 leader_election.cc:341] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.604032 18355 leader_election.cc:302] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:48.605449 18441 raft_consensus.cc:2781] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.605799 18441 raft_consensus.cc:482] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.606108 18441 raft_consensus.cc:3037] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.606910 18433 consensus_queue.cc:260] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.613131 18433 ts_tablet_manager.cc:1398] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20240502 01:49:48.614017 18433 tablet_bootstrap.cc:492] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.614387 18441 raft_consensus.cc:504] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.617295 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
W20240502 01:49:48.618639 18356 leader_election.cc:341] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.619887 18433 tablet_bootstrap.cc:654] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.620654 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:48.621227 18315 raft_consensus.cc:3037] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.624433 18481 raft_consensus.cc:482] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.624910 18481 raft_consensus.cc:504] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.627213 18481 leader_election.cc:288] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.628150 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a06b4fbd949e490a4e206fa313777" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.631460 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a06b4fbd949e490a4e206fa313777" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.632403 18392 raft_consensus.cc:2379] T 7b4a06b4fbd949e490a4e206fa313777 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:48.633664 18203 leader_election.cc:302] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.634558 18451 raft_consensus.cc:3037] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.639309 18491 raft_consensus.cc:482] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.639674 18491 raft_consensus.cc:504] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.642689 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa7efd1cfbd4c81bbcde777d6eeb082" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.642978 18451 raft_consensus.cc:2726] T 7b4a06b4fbd949e490a4e206fa313777 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.643370 18316 raft_consensus.cc:2454] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:48.644762 18201 leader_election.cc:302] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:48.646819 18438 raft_consensus.cc:482] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.647284 18438 raft_consensus.cc:504] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.648054 18481 raft_consensus.cc:2781] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.648483 18481 raft_consensus.cc:482] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.648864 18481 raft_consensus.cc:3037] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.649286 18438 leader_election.cc:288] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.650851 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.651337 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa7efd1cfbd4c81bbcde777d6eeb082" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:48.652401 18356 leader_election.cc:341] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:48.652710 18203 leader_election.cc:341] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.653358 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.653898 18491 leader_election.cc:288] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:48.654701 18355 leader_election.cc:341] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.655167 18355 leader_election.cc:302] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:48.656296 18438 raft_consensus.cc:2726] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.660588 18315 raft_consensus.cc:2454] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:48.661779 18355 leader_election.cc:302] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:48.662621 18438 raft_consensus.cc:2781] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:48.680850 18441 leader_election.cc:288] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.694139 18475 raft_consensus.cc:482] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.694756 18475 raft_consensus.cc:504] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.696969 18475 leader_election.cc:288] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.697839 18438 raft_consensus.cc:686] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:48.698194 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.698787 18438 consensus_queue.cc:237] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:48.699946 18280 leader_election.cc:341] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.701382 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.696985 18493 consensus_queue.cc:1035] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 01:49:48.702129 18392 raft_consensus.cc:2454] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:48.702586 18435 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.701222 18481 raft_consensus.cc:504] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.703056 18435 ts_tablet_manager.cc:1367] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.108s user 0.016s sys 0.034s
I20240502 01:49:48.703378 18280 leader_election.cc:302] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:48.704485 18475 raft_consensus.cc:2781] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.704821 18475 raft_consensus.cc:482] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.705137 18475 raft_consensus.cc:3037] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.705807 18435 raft_consensus.cc:348] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.706532 18435 raft_consensus.cc:374] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.706804 18435 raft_consensus.cc:729] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.707398 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa7efd1cfbd4c81bbcde777d6eeb082" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:48.707531 18435 consensus_queue.cc:260] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:48.708544 18203 leader_election.cc:341] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.713188 18435 ts_tablet_manager.cc:1398] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240502 01:49:48.715042 18493 consensus_queue.cc:1035] T da54a35564ce4389a2395ea0832a968b P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:48.715047 18435 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.711048 18481 leader_election.cc:288] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:48.718271 18201 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 73605.8us
I20240502 01:49:48.720885 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa7efd1cfbd4c81bbcde777d6eeb082" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:48.722013 18316 raft_consensus.cc:3037] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.723057 18433 tablet_bootstrap.cc:492] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.723495 18433 ts_tablet_manager.cc:1367] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.033s
I20240502 01:49:48.719652 18481 raft_consensus.cc:482] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.727802 18481 raft_consensus.cc:504] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.729923 18435 tablet_bootstrap.cc:654] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.730804 18475 raft_consensus.cc:504] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.731282 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.732148 18392 raft_consensus.cc:2454] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:48.732926 18434 tablet_bootstrap.cc:492] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.733384 18203 leader_election.cc:302] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:48.734228 18316 raft_consensus.cc:2454] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:48.734378 18492 raft_consensus.cc:2781] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.729959 18481 leader_election.cc:288] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.734745 18433 raft_consensus.cc:348] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.735191 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:48.735463 18433 raft_consensus.cc:374] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.734793 18492 raft_consensus.cc:482] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.735543 18434 ts_tablet_manager.cc:1367] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.141s user 0.053s sys 0.060s
I20240502 01:49:48.736073 18492 raft_consensus.cc:3037] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.736321 18433 raft_consensus.cc:729] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.737180 18475 leader_election.cc:288] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.737089 18433 consensus_queue.cc:260] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.738678 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:48.738940 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.738845 18434 raft_consensus.cc:348] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:48.739419 18433 ts_tablet_manager.cc:1398] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20240502 01:49:48.739912 18493 raft_consensus.cc:482] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 01:49:48.740118 18280 leader_election.cc:341] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.740360 18433 tablet_bootstrap.cc:492] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.740502 18392 raft_consensus.cc:3037] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:48.743331 18201 leader_election.cc:341] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.744321 18493 raft_consensus.cc:504] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.746003 18433 tablet_bootstrap.cc:654] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.746495 18493 leader_election.cc:288] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.748121 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b8a888ea634dcb802badc6a669ecc7" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.744130 18434 raft_consensus.cc:374] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.748937 18434 raft_consensus.cc:729] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
W20240502 01:49:48.749524 18355 leader_election.cc:341] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.749752 18434 consensus_queue.cc:260] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:48.750411 18435 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.750968 18435 ts_tablet_manager.cc:1367] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.000s
I20240502 01:49:48.752084 18434 ts_tablet_manager.cc:1398] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20240502 01:49:48.753048 18434 tablet_bootstrap.cc:492] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.753224 18435 raft_consensus.cc:348] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.753875 18435 raft_consensus.cc:374] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.754122 18435 raft_consensus.cc:729] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.754719 18435 consensus_queue.cc:260] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.756453 18493 raft_consensus.cc:482] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.756932 18435 ts_tablet_manager.cc:1398] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:49:48.757272 18493 raft_consensus.cc:504] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.757773 18435 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.759070 18434 tablet_bootstrap.cc:654] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.764022 18435 tablet_bootstrap.cc:654] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.764161 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a18b76f334dc281187e58de8d90e4" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.765055 18481 raft_consensus.cc:482] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.765571 18481 raft_consensus.cc:504] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.766954 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a18b76f334dc281187e58de8d90e4" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.768697 18433 tablet_bootstrap.cc:492] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.769151 18433 ts_tablet_manager.cc:1367] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20240502 01:49:48.769580 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.767657 18481 leader_election.cc:288] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.771522 18433 raft_consensus.cc:348] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.772207 18433 raft_consensus.cc:374] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 01:49:48.772320 18355 leader_election.cc:341] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.772486 18433 raft_consensus.cc:729] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.773113 18433 consensus_queue.cc:260] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.775324 18433 ts_tablet_manager.cc:1398] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:49:48.776225 18433 tablet_bootstrap.cc:492] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.777824 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3eec00736204e3fa0dcf0797f7b0d75" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.778712 18313 raft_consensus.cc:2379] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:48.779996 18201 leader_election.cc:302] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.780773 18481 raft_consensus.cc:3037] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.781304 18433 tablet_bootstrap.cc:654] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:48.781644 18356 leader_election.cc:341] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.782069 18356 leader_election.cc:302] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:48.782826 18493 leader_election.cc:288] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.782941 18494 raft_consensus.cc:2726] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.783908 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b8a888ea634dcb802badc6a669ecc7" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.784567 18238 raft_consensus.cc:2454] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:48.786063 18435 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.786516 18435 ts_tablet_manager.cc:1367] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20240502 01:49:48.788657 18481 raft_consensus.cc:2726] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.788894 18435 raft_consensus.cc:348] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.780750 18434 tablet_bootstrap.cc:492] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.789589 18435 raft_consensus.cc:374] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.789921 18435 raft_consensus.cc:729] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.790086 18434 ts_tablet_manager.cc:1367] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.003s
I20240502 01:49:48.790488 18492 raft_consensus.cc:504] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.791997 18435 consensus_queue.cc:260] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.793908 18435 ts_tablet_manager.cc:1398] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:49:48.794682 18435 tablet_bootstrap.cc:492] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.796989 18434 raft_consensus.cc:348] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.797720 18434 raft_consensus.cc:374] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.798032 18434 raft_consensus.cc:729] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.798653 18434 consensus_queue.cc:260] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.799635 18435 tablet_bootstrap.cc:654] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.792200 18356 leader_election.cc:302] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:48.800660 18493 raft_consensus.cc:2781] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.801051 18493 raft_consensus.cc:482] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.801414 18493 raft_consensus.cc:3037] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.800854 18434 ts_tablet_manager.cc:1398] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:49:48.802913 18434 tablet_bootstrap.cc:492] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.803489 18433 tablet_bootstrap.cc:492] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.804011 18433 ts_tablet_manager.cc:1367] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.004s
I20240502 01:49:48.803180 18131 catalog_manager.cc:5484] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:48.804788 18201 leader_election.cc:302] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.808562 18492 leader_election.cc:288] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.809096 18492 raft_consensus.cc:2781] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:48.806102 18392 raft_consensus.cc:2454] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:48.809619 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:48.806557 18433 raft_consensus.cc:348] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.810199 18391 raft_consensus.cc:3037] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.810253 18280 leader_election.cc:302] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:48.811609 18475 raft_consensus.cc:2781] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:48.812108 18475 raft_consensus.cc:686] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:48.812906 18475 consensus_queue.cc:237] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.813938 18492 raft_consensus.cc:686] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:48.815016 18492 consensus_queue.cc:237] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.819988 18391 raft_consensus.cc:2454] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:48.810216 18433 raft_consensus.cc:374] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.821187 18433 raft_consensus.cc:729] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.821851 18433 consensus_queue.cc:260] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.824044 18433 ts_tablet_manager.cc:1398] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240502 01:49:48.824239 18203 leader_election.cc:302] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:48.825035 18433 tablet_bootstrap.cc:492] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.822952 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:48.825233 18481 raft_consensus.cc:2781] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:48.825757 18481 raft_consensus.cc:686] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
W20240502 01:49:48.826205 18201 leader_election.cc:341] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.826608 18481 consensus_queue.cc:237] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.808045 18434 tablet_bootstrap.cc:654] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.834714 18433 tablet_bootstrap.cc:654] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.842975 18435 tablet_bootstrap.cc:492] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.842646 18493 raft_consensus.cc:504] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.843446 18435 ts_tablet_manager.cc:1367] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.021s
I20240502 01:49:48.843672 18442 consensus_queue.cc:1035] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:48.846428 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b8a888ea634dcb802badc6a669ecc7" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:48.847045 18239 raft_consensus.cc:3037] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.846947 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b8a888ea634dcb802badc6a669ecc7" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:48.847591 18316 raft_consensus.cc:3037] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.852974 18433 tablet_bootstrap.cc:492] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:48.853519 18433 ts_tablet_manager.cc:1367] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.008s
I20240502 01:49:48.854357 18493 leader_election.cc:288] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.855998 18435 raft_consensus.cc:348] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.856881 18435 raft_consensus.cc:374] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.858472 18316 raft_consensus.cc:2454] T b7b8a888ea634dcb802badc6a669ecc7 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:48.862515 18435 raft_consensus.cc:729] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.858809 18433 raft_consensus.cc:348] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.862994 18433 raft_consensus.cc:374] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.860358 18239 raft_consensus.cc:2454] T b7b8a888ea634dcb802badc6a669ecc7 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:48.863255 18435 consensus_queue.cc:260] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.864742 18356 leader_election.cc:302] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:48.863283 18433 raft_consensus.cc:729] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.865872 18438 raft_consensus.cc:2781] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:48.866415 18438 raft_consensus.cc:686] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:48.865927 18433 consensus_queue.cc:260] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.867250 18438 consensus_queue.cc:237] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.869117 18435 ts_tablet_manager.cc:1398] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20240502 01:49:48.870066 18435 tablet_bootstrap.cc:492] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.875618 18433 ts_tablet_manager.cc:1398] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.022s user 0.007s sys 0.002s
I20240502 01:49:48.876821 18433 tablet_bootstrap.cc:492] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:48.891505 18492 raft_consensus.cc:482] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.892060 18492 raft_consensus.cc:504] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.886934 18130 catalog_manager.cc:5484] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:48.895327 18492 leader_election.cc:288] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.898593 18435 tablet_bootstrap.cc:654] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.899995 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5090d855c6864a3d904fe1141d503d9d" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:48.898555 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5090d855c6864a3d904fe1141d503d9d" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:48.901697 18201 leader_election.cc:341] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:48.902556 18203 leader_election.cc:341] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:48.903015 18203 leader_election.cc:302] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:48.903816 18492 raft_consensus.cc:2726] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:48.916735 18434 tablet_bootstrap.cc:492] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:48.917291 18434 ts_tablet_manager.cc:1367] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.005s
I20240502 01:49:48.920558 18434 raft_consensus.cc:348] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.921280 18434 raft_consensus.cc:374] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.921576 18434 raft_consensus.cc:729] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.922238 18434 consensus_queue.cc:260] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.924978 18434 ts_tablet_manager.cc:1398] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:49:48.925940 18434 tablet_bootstrap.cc:492] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:48.931016 18435 tablet_bootstrap.cc:492] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:48.930649 18475 consensus_queue.cc:1035] T 7b4a06b4fbd949e490a4e206fa313777 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:48.933959 18435 ts_tablet_manager.cc:1367] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.005s
I20240502 01:49:48.934751 18433 tablet_bootstrap.cc:654] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.937652 18435 raft_consensus.cc:348] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.938633 18435 raft_consensus.cc:374] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:48.938974 18435 raft_consensus.cc:729] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:48.940040 18435 consensus_queue.cc:260] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.943195 18435 ts_tablet_manager.cc:1398] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:49:48.944222 18435 tablet_bootstrap.cc:492] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:48.944293 18131 catalog_manager.cc:5484] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:48.949815 18442 raft_consensus.cc:482] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.952643 18442 raft_consensus.cc:504] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:48.953362 18434 tablet_bootstrap.cc:654] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.954694 18442 leader_election.cc:288] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775), 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:49:48.956724 18442 raft_consensus.cc:482] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.960626 18435 tablet_bootstrap.cc:654] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:48.960067 18442 raft_consensus.cc:504] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.963572 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a75f4d1c0c746f3ba51960f4f4e4f55" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.964715 18442 leader_election.cc:288] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:48.964314 18392 raft_consensus.cc:2454] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:48.965343 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a75f4d1c0c746f3ba51960f4f4e4f55" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.967152 18316 raft_consensus.cc:3037] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.966956 18132 catalog_manager.cc:5484] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:48.972321 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba72fe24abe4fe893e60d48e6679580" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:48.984874 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba72fe24abe4fe893e60d48e6679580" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:48.988807 18280 leader_election.cc:302] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:48.992902 18515 raft_consensus.cc:2781] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:48.993408 18515 raft_consensus.cc:482] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:48.993767 18515 raft_consensus.cc:3037] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:48.994884 18438 raft_consensus.cc:482] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:48.995422 18438 raft_consensus.cc:504] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:48.997624 18438 leader_election.cc:288] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:48.996927 18130 catalog_manager.cc:5484] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:48.999511 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa7efd1cfbd4c81bbcde777d6eeb082" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.000342 18315 raft_consensus.cc:2379] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:49.000214 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa7efd1cfbd4c81bbcde777d6eeb082" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.002104 18355 leader_election.cc:302] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.002249 18515 raft_consensus.cc:504] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:49.003288 18438 raft_consensus.cc:3037] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:49.006193 18280 leader_election.cc:341] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.012598 18515 leader_election.cc:288] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775), 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:49:49.009676 18280 leader_election.cc:302] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:49.026825 18435 tablet_bootstrap.cc:492] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.027356 18435 ts_tablet_manager.cc:1367] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.008s
I20240502 01:49:49.036703 18435 raft_consensus.cc:348] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:49.037446 18435 raft_consensus.cc:374] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.037734 18435 raft_consensus.cc:729] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.038750 18435 consensus_queue.cc:260] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:49.040251 18493 raft_consensus.cc:482] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.040762 18493 raft_consensus.cc:504] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.041193 18438 raft_consensus.cc:2726] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.042484 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a75f4d1c0c746f3ba51960f4f4e4f55" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.042834 18493 leader_election.cc:288] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.043152 18391 raft_consensus.cc:3037] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.043985 18515 raft_consensus.cc:2726] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.044529 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.040787 18433 tablet_bootstrap.cc:492] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.045956 18433 ts_tablet_manager.cc:1367] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.169s user 0.015s sys 0.004s
I20240502 01:49:49.046766 18493 raft_consensus.cc:482] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.047243 18493 raft_consensus.cc:504] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:49.047780 18356 leader_election.cc:341] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.048872 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a75f4d1c0c746f3ba51960f4f4e4f55" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.049448 18238 raft_consensus.cc:3037] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.050684 18433 raft_consensus.cc:348] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.051357 18433 raft_consensus.cc:374] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.051614 18433 raft_consensus.cc:729] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.052202 18433 consensus_queue.cc:260] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.054577 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.055321 18433 ts_tablet_manager.cc:1398] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 01:49:49.055816 18493 leader_election.cc:288] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:49.056605 18355 leader_election.cc:341] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.057068 18355 leader_election.cc:302] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.057715 18493 raft_consensus.cc:2726] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.058837 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7889fa2a16487d8b767d51bb21abf2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.059275 18433 tablet_bootstrap.cc:492] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.058954 18391 raft_consensus.cc:2454] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.064039 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7889fa2a16487d8b767d51bb21abf2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.064388 18435 ts_tablet_manager.cc:1398] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.037s user 0.003s sys 0.005s
I20240502 01:49:49.064832 18239 raft_consensus.cc:2454] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:49.066033 18356 leader_election.cc:302] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
W20240502 01:49:49.067636 18355 leader_election.cc:341] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.069062 18433 tablet_bootstrap.cc:654] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.069214 18280 leader_election.cc:302] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:49.070178 18515 raft_consensus.cc:2781] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.070683 18515 raft_consensus.cc:686] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:49.072090 18434 tablet_bootstrap.cc:492] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.071518 18515 consensus_queue.cc:237] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } }
I20240502 01:49:49.067631 18438 raft_consensus.cc:2781] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.072706 18434 ts_tablet_manager.cc:1367] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.147s user 0.027s sys 0.014s
I20240502 01:49:49.072827 18438 raft_consensus.cc:482] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.073319 18438 raft_consensus.cc:3037] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.075517 18434 raft_consensus.cc:348] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.075968 18492 raft_consensus.cc:482] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.076401 18434 raft_consensus.cc:374] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.076776 18434 raft_consensus.cc:729] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.076520 18492 raft_consensus.cc:504] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.077626 18434 consensus_queue.cc:260] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.078272 18238 raft_consensus.cc:2454] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.078559 18492 leader_election.cc:288] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.080152 18434 ts_tablet_manager.cc:1398] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:49:49.080492 18438 raft_consensus.cc:504] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.081221 18434 tablet_bootstrap.cc:492] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.083317 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.083772 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7889fa2a16487d8b767d51bb21abf2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.083897 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fdfbadfdb3472782d6a189ffbbe112" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.084457 18239 raft_consensus.cc:3037] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.084131 18316 raft_consensus.cc:2379] T 47fdfbadfdb3472782d6a189ffbbe112 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:49.085698 18435 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.086886 18201 leader_election.cc:302] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:49.088198 18492 raft_consensus.cc:3037] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.095921 18239 raft_consensus.cc:2454] T af7889fa2a16487d8b767d51bb21abf2 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:49.095721 18130 catalog_manager.cc:5484] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.097209 18356 leader_election.cc:302] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:49.098088 18493 raft_consensus.cc:2781] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.098574 18493 raft_consensus.cc:686] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:49.098793 18435 tablet_bootstrap.cc:654] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.099390 18492 raft_consensus.cc:2726] T 47fdfbadfdb3472782d6a189ffbbe112 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.098006 18434 tablet_bootstrap.cc:654] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.099406 18493 consensus_queue.cc:237] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.105182 18434 tablet_bootstrap.cc:492] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.105727 18434 ts_tablet_manager.cc:1367] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.002s
I20240502 01:49:49.106194 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7889fa2a16487d8b767d51bb21abf2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:49.106786 18316 raft_consensus.cc:3037] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.109773 18434 raft_consensus.cc:348] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.110451 18434 raft_consensus.cc:374] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.110697 18434 raft_consensus.cc:729] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.111375 18434 consensus_queue.cc:260] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.112628 18130 catalog_manager.cc:5484] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:49.113692 18434 ts_tablet_manager.cc:1398] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:49:49.114665 18434 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.117319 18492 raft_consensus.cc:482] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.117789 18492 raft_consensus.cc:504] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.119047 18316 raft_consensus.cc:2454] T af7889fa2a16487d8b767d51bb21abf2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:49.121012 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a18b76f334dc281187e58de8d90e4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.121654 18315 raft_consensus.cc:2454] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:49.123356 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a18b76f334dc281187e58de8d90e4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.123473 18438 leader_election.cc:288] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.124059 18391 raft_consensus.cc:2454] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:49.125710 18201 leader_election.cc:302] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:49.126610 18481 raft_consensus.cc:2781] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.126888 18481 raft_consensus.cc:482] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.127133 18481 raft_consensus.cc:3037] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.128613 18433 tablet_bootstrap.cc:492] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.119710 18492 leader_election.cc:288] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.132176 18435 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.132629 18435 ts_tablet_manager.cc:1367] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.007s
I20240502 01:49:49.133172 18434 tablet_bootstrap.cc:654] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.134164 18481 raft_consensus.cc:504] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.135092 18435 raft_consensus.cc:348] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.135769 18435 raft_consensus.cc:374] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.136068 18435 raft_consensus.cc:729] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.129052 18433 ts_tablet_manager.cc:1367] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.070s user 0.017s sys 0.002s
I20240502 01:49:49.136943 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a18b76f334dc281187e58de8d90e4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:49.137598 18316 raft_consensus.cc:3037] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.138965 18433 raft_consensus.cc:348] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.139456 18433 raft_consensus.cc:374] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.139654 18433 raft_consensus.cc:729] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.139858 18434 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.140316 18434 ts_tablet_manager.cc:1367] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.002s
I20240502 01:49:49.140203 18433 consensus_queue.cc:260] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.142006 18433 ts_tablet_manager.cc:1398] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:49:49.142495 18435 consensus_queue.cc:260] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.142671 18434 raft_consensus.cc:348] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.143357 18434 raft_consensus.cc:374] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.143617 18434 raft_consensus.cc:729] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.144131 18316 raft_consensus.cc:2454] T 231a18b76f334dc281187e58de8d90e4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:49.144318 18435 ts_tablet_manager.cc:1398] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:49:49.144224 18434 consensus_queue.cc:260] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.145421 18201 leader_election.cc:302] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:49.146229 18434 ts_tablet_manager.cc:1398] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:49:49.146229 18492 raft_consensus.cc:2781] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.146806 18492 raft_consensus.cc:686] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:49.147338 18434 tablet_bootstrap.cc:492] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.147650 18433 tablet_bootstrap.cc:492] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.147570 18492 consensus_queue.cc:237] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.152395 18434 tablet_bootstrap.cc:654] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.153836 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a18b76f334dc281187e58de8d90e4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.154443 18391 raft_consensus.cc:3037] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.161799 18391 raft_consensus.cc:2454] T 231a18b76f334dc281187e58de8d90e4 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:49.163357 18481 leader_election.cc:288] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.165643 18435 tablet_bootstrap.cc:492] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.166359 18434 tablet_bootstrap.cc:492] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.166711 18434 ts_tablet_manager.cc:1367] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240502 01:49:49.168606 18433 tablet_bootstrap.cc:654] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.168975 18434 raft_consensus.cc:348] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.169639 18434 raft_consensus.cc:374] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.169900 18434 raft_consensus.cc:729] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.170580 18434 consensus_queue.cc:260] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.174335 18433 tablet_bootstrap.cc:492] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.174829 18433 ts_tablet_manager.cc:1367] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.007s
I20240502 01:49:49.175559 18434 ts_tablet_manager.cc:1398] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 01:49:49.176441 18434 tablet_bootstrap.cc:492] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.177402 18433 raft_consensus.cc:348] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.178148 18433 raft_consensus.cc:374] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.178721 18433 raft_consensus.cc:729] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.178279 18132 catalog_manager.cc:5484] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.179486 18433 consensus_queue.cc:260] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.181674 18433 ts_tablet_manager.cc:1398] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240502 01:49:49.182343 18434 tablet_bootstrap.cc:654] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.182623 18433 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.188078 18433 tablet_bootstrap.cc:654] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.188874 18438 raft_consensus.cc:482] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.189379 18438 raft_consensus.cc:504] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.191423 18438 leader_election.cc:288] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.192940 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf23c8fedfe4430bbb092f9e927d55e5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:49.194325 18356 leader_election.cc:341] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.196159 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf23c8fedfe4430bbb092f9e927d55e5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.196801 18316 raft_consensus.cc:2454] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:49.198081 18435 tablet_bootstrap.cc:654] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.198156 18355 leader_election.cc:302] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.199388 18438 raft_consensus.cc:2781] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.199815 18438 raft_consensus.cc:482] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.200217 18438 raft_consensus.cc:3037] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.204218 18493 raft_consensus.cc:482] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.204761 18493 raft_consensus.cc:504] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.207028 18493 leader_election.cc:288] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.208794 18435 tablet_bootstrap.cc:492] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.208949 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.209347 18435 ts_tablet_manager.cc:1367] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.000s
W20240502 01:49:49.210603 18356 leader_election.cc:341] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.211050 18438 raft_consensus.cc:504] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.211818 18435 raft_consensus.cc:348] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.212435 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.212692 18435 raft_consensus.cc:374] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.213037 18435 raft_consensus.cc:729] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.213718 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf23c8fedfe4430bbb092f9e927d55e5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.213627 18435 consensus_queue.cc:260] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:49.215093 18356 leader_election.cc:341] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.216181 18438 leader_election.cc:288] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.216342 18435 ts_tablet_manager.cc:1398] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
W20240502 01:49:49.214159 18355 leader_election.cc:341] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.217837 18355 leader_election.cc:302] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.218338 18435 tablet_bootstrap.cc:492] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.218793 18434 tablet_bootstrap.cc:492] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.219236 18434 ts_tablet_manager.cc:1367] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.043s user 0.023s sys 0.016s
I20240502 01:49:49.219729 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf23c8fedfe4430bbb092f9e927d55e5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:49.220309 18316 raft_consensus.cc:3037] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.221652 18434 raft_consensus.cc:348] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.222321 18434 raft_consensus.cc:374] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.222566 18434 raft_consensus.cc:729] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.223254 18434 consensus_queue.cc:260] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.224126 18435 tablet_bootstrap.cc:654] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.227332 18493 raft_consensus.cc:2726] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.227954 18493 raft_consensus.cc:482] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.228420 18493 raft_consensus.cc:504] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.230297 18493 leader_election.cc:288] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.231166 18434 ts_tablet_manager.cc:1398] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20240502 01:49:49.231776 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.232287 18434 tablet_bootstrap.cc:492] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
W20240502 01:49:49.233356 18356 leader_election.cc:341] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.234797 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:49.236156 18355 leader_election.cc:341] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.236598 18355 leader_election.cc:302] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.237092 18316 raft_consensus.cc:2454] T cf23c8fedfe4430bbb092f9e927d55e5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:49.204766 18433 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.238147 18433 ts_tablet_manager.cc:1367] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.016s
I20240502 01:49:49.238221 18355 leader_election.cc:302] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.239315 18493 raft_consensus.cc:2726] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.239518 18438 raft_consensus.cc:2781] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.240092 18438 raft_consensus.cc:686] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:49.240839 18433 raft_consensus.cc:348] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.240914 18438 consensus_queue.cc:237] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.241607 18433 raft_consensus.cc:374] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.241681 18434 tablet_bootstrap.cc:654] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.241967 18433 raft_consensus.cc:729] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.242686 18435 tablet_bootstrap.cc:492] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.242611 18433 consensus_queue.cc:260] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.243183 18435 ts_tablet_manager.cc:1367] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.013s
I20240502 01:49:49.245860 18435 raft_consensus.cc:348] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.246645 18435 raft_consensus.cc:374] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.246974 18435 raft_consensus.cc:729] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.247684 18435 consensus_queue.cc:260] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.248481 18481 consensus_queue.cc:1035] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.250064 18435 ts_tablet_manager.cc:1398] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 01:49:49.254575 18433 ts_tablet_manager.cc:1398] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
I20240502 01:49:49.255635 18433 tablet_bootstrap.cc:492] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.261188 18433 tablet_bootstrap.cc:654] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.251037 18435 tablet_bootstrap.cc:492] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.266583 18435 tablet_bootstrap.cc:654] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.267071 18451 consensus_queue.cc:1035] T 9fa7efd1cfbd4c81bbcde777d6eeb082 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.275400 18481 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.275628 18494 consensus_queue.cc:1035] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:49:49.269021 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:49.288928 18515 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:49.299046 18434 tablet_bootstrap.cc:492] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.299541 18434 ts_tablet_manager.cc:1367] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.020s
I20240502 01:49:49.301975 18434 raft_consensus.cc:348] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.302687 18434 raft_consensus.cc:374] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.302953 18434 raft_consensus.cc:729] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.303539 18434 consensus_queue.cc:260] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.308679 18434 ts_tablet_manager.cc:1398] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240502 01:49:49.309629 18434 tablet_bootstrap.cc:492] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.309902 18133 catalog_manager.cc:5484] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:49.313700 18433 tablet_bootstrap.cc:492] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.314174 18433 ts_tablet_manager.cc:1367] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.008s
I20240502 01:49:49.314785 18434 tablet_bootstrap.cc:654] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.318809 18433 raft_consensus.cc:348] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.320603 18433 raft_consensus.cc:374] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.320901 18433 raft_consensus.cc:729] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.321593 18433 consensus_queue.cc:260] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.322611 18493 consensus_queue.cc:1035] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.323288 18441 consensus_queue.cc:1035] T 47fdfbadfdb3472782d6a189ffbbe112 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.323509 18315 raft_consensus.cc:3037] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.324625 18433 ts_tablet_manager.cc:1398] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20240502 01:49:49.325691 18433 tablet_bootstrap.cc:492] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.328423 18493 raft_consensus.cc:482] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.328948 18493 raft_consensus.cc:504] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.330885 18433 tablet_bootstrap.cc:654] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.331455 18493 leader_election.cc:288] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.336474 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.337728 18435 tablet_bootstrap.cc:492] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.338184 18435 ts_tablet_manager.cc:1367] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.011s
I20240502 01:49:49.340735 18435 raft_consensus.cc:348] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.341400 18435 raft_consensus.cc:374] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.341667 18435 raft_consensus.cc:729] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.345328 18435 consensus_queue.cc:260] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.346262 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.346958 18314 raft_consensus.cc:2454] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
W20240502 01:49:49.348106 18356 leader_election.cc:341] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.347163 18435 ts_tablet_manager.cc:1398] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 01:49:49.350101 18435 tablet_bootstrap.cc:492] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.355284 18434 tablet_bootstrap.cc:492] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.355798 18434 ts_tablet_manager.cc:1367] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.000s
I20240502 01:49:49.361584 18475 raft_consensus.cc:482] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.362134 18475 raft_consensus.cc:504] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.354832 18515 raft_consensus.cc:482] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.364290 18515 raft_consensus.cc:504] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.366072 18475 leader_election.cc:288] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.366488 18515 leader_election.cc:288] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.367467 18355 leader_election.cc:302] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.370200 18438 raft_consensus.cc:2781] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.370589 18438 raft_consensus.cc:482] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.370947 18438 raft_consensus.cc:3037] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.384910 18451 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.385298 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.387254 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7943deee0b2f49cf8a391e59d9676aa1" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.388566 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7943deee0b2f49cf8a391e59d9676aa1" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.389273 18235 raft_consensus.cc:2454] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:49.393687 18280 leader_election.cc:302] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:49.395619 18441 consensus_queue.cc:1035] T b7b8a888ea634dcb802badc6a669ecc7 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.407724 18515 raft_consensus.cc:2781] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.408262 18515 raft_consensus.cc:482] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.408851 18515 raft_consensus.cc:3037] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.413100 18475 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:49:49.422212 18280 leader_election.cc:341] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.427006 18438 raft_consensus.cc:504] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.429260 18438 leader_election.cc:288] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.441187 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.446591 18547 raft_consensus.cc:482] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.447132 18547 raft_consensus.cc:504] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.449184 18547 leader_election.cc:288] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:49.449965 18356 leader_election.cc:341] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.457420 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:49.458043 18316 raft_consensus.cc:3037] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.458029 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.462793 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.467020 18515 raft_consensus.cc:504] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.456468 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caecb89696f549ac8c4ab781e42472b1" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:49.470597 18201 leader_election.cc:341] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.472805 18435 tablet_bootstrap.cc:654] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:49.474452 18203 leader_election.cc:341] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.477643 18433 tablet_bootstrap.cc:492] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.478119 18433 ts_tablet_manager.cc:1367] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.153s user 0.049s sys 0.090s
I20240502 01:49:49.480413 18434 raft_consensus.cc:348] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.467574 18316 raft_consensus.cc:2454] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:49.492002 18355 leader_election.cc:302] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.493327 18438 raft_consensus.cc:2781] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.494019 18515 leader_election.cc:288] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.493837 18438 raft_consensus.cc:686] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:49.494982 18438 consensus_queue.cc:237] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.476934 18203 leader_election.cc:302] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:49.481973 18280 leader_election.cc:302] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:49.486624 18433 raft_consensus.cc:348] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.499060 18433 raft_consensus.cc:374] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.499303 18433 raft_consensus.cc:729] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.499799 18433 consensus_queue.cc:260] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.496764 18434 raft_consensus.cc:374] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.502353 18433 ts_tablet_manager.cc:1398] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20240502 01:49:49.503293 18433 tablet_bootstrap.cc:492] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.500641 18434 raft_consensus.cc:729] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.505533 18547 raft_consensus.cc:2726] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.506579 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7943deee0b2f49cf8a391e59d9676aa1" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.506661 18434 consensus_queue.cc:260] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.507215 18392 raft_consensus.cc:3037] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.498855 18515 raft_consensus.cc:2726] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:49:49.510890 18434 ts_tablet_manager.cc:1398] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.155s user 0.011s sys 0.003s
I20240502 01:49:49.511776 18434 tablet_bootstrap.cc:492] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.513756 18433 tablet_bootstrap.cc:654] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.516081 18547 raft_consensus.cc:482] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.516554 18547 raft_consensus.cc:504] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.518460 18547 leader_election.cc:288] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.519281 18392 raft_consensus.cc:2454] T 7943deee0b2f49cf8a391e59d9676aa1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.520879 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7943deee0b2f49cf8a391e59d9676aa1" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.522238 18280 leader_election.cc:302] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:49.522469 18236 raft_consensus.cc:3037] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.523721 18515 raft_consensus.cc:2781] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.524221 18515 raft_consensus.cc:686] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:49.524595 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.525039 18515 consensus_queue.cc:237] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:49.526077 18201 leader_election.cc:341] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.526769 18547 raft_consensus.cc:482] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.527247 18547 raft_consensus.cc:504] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.523363 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.530437 18392 raft_consensus.cc:2454] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:49.532337 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e09bfc9fdb46af9dd702b4c831e287" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.532548 18547 leader_election.cc:288] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:49.533627 18201 leader_election.cc:341] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.534600 18435 tablet_bootstrap.cc:492] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.535023 18435 ts_tablet_manager.cc:1367] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.185s user 0.006s sys 0.014s
I20240502 01:49:49.537281 18435 raft_consensus.cc:348] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.537935 18435 raft_consensus.cc:374] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.538199 18435 raft_consensus.cc:729] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.539673 18435 consensus_queue.cc:260] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.541973 18435 ts_tablet_manager.cc:1398] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:49:49.542716 18434 tablet_bootstrap.cc:654] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.544163 18203 leader_election.cc:302] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.544966 18547 raft_consensus.cc:2781] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.545528 18547 raft_consensus.cc:482] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.545850 18547 raft_consensus.cc:3037] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.548147 18236 raft_consensus.cc:2454] T 7943deee0b2f49cf8a391e59d9676aa1 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.553582 18547 raft_consensus.cc:504] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.555517 18547 leader_election.cc:288] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.556313 18130 catalog_manager.cc:5484] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:49.543236 18435 tablet_bootstrap.cc:492] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.558388 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
W20240502 01:49:49.559635 18201 leader_election.cc:341] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.563469 18435 tablet_bootstrap.cc:654] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.565937 18475 consensus_queue.cc:1035] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.568162 18433 tablet_bootstrap.cc:492] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.568601 18433 ts_tablet_manager.cc:1367] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.007s
I20240502 01:49:49.569523 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.570083 18392 raft_consensus.cc:3037] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.572585 18434 tablet_bootstrap.cc:492] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.573040 18434 ts_tablet_manager.cc:1367] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.006s
I20240502 01:49:49.573463 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e09bfc9fdb46af9dd702b4c831e287" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.574219 18391 raft_consensus.cc:2454] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:49.575345 18203 leader_election.cc:302] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.576443 18547 raft_consensus.cc:2781] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.576797 18547 raft_consensus.cc:482] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.577028 18475 raft_consensus.cc:482] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.577116 18547 raft_consensus.cc:3037] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.577522 18475 raft_consensus.cc:504] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.580987 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a9d50e756d4871a8789ea75c671cca" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.581465 18475 leader_election.cc:288] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:49.582304 18280 leader_election.cc:341] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.584332 18130 catalog_manager.cc:5484] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.594197 18438 consensus_queue.cc:1035] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.585724 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a9d50e756d4871a8789ea75c671cca" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.595105 18391 raft_consensus.cc:2454] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:49.586165 18547 raft_consensus.cc:504] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.590181 18513 consensus_queue.cc:1035] T 9a75f4d1c0c746f3ba51960f4f4e4f55 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:49.591477 18434 raft_consensus.cc:348] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.596328 18434 raft_consensus.cc:374] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.592631 18433 raft_consensus.cc:348] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.596809 18433 raft_consensus.cc:374] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.597218 18433 raft_consensus.cc:729] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.597744 18547 leader_election.cc:288] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.596194 18280 leader_election.cc:302] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.592787 18435 tablet_bootstrap.cc:492] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.598840 18513 raft_consensus.cc:2781] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.599169 18513 raft_consensus.cc:482] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.599480 18513 raft_consensus.cc:3037] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.600008 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e09bfc9fdb46af9dd702b4c831e287" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
W20240502 01:49:49.601284 18201 leader_election.cc:341] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.602013 18433 consensus_queue.cc:260] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.604182 18433 ts_tablet_manager.cc:1398] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.035s user 0.005s sys 0.002s
I20240502 01:49:49.596776 18434 raft_consensus.cc:729] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.606824 18433 tablet_bootstrap.cc:492] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.608968 18392 raft_consensus.cc:2454] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:49.610016 18203 leader_election.cc:302] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.611204 18547 raft_consensus.cc:2781] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.611719 18547 raft_consensus.cc:686] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:49.612500 18513 raft_consensus.cc:504] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.612591 18547 consensus_queue.cc:237] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.613462 18435 ts_tablet_manager.cc:1367] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.071s user 0.018s sys 0.009s
I20240502 01:49:49.617879 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e09bfc9fdb46af9dd702b4c831e287" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.618476 18391 raft_consensus.cc:3037] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.619951 18435 raft_consensus.cc:348] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.620620 18435 raft_consensus.cc:374] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.620864 18435 raft_consensus.cc:729] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.621977 18513 leader_election.cc:288] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.622727 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a9d50e756d4871a8789ea75c671cca" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.623514 18392 raft_consensus.cc:3037] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.623720 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a9d50e756d4871a8789ea75c671cca" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.605634 18434 consensus_queue.cc:260] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.623957 18435 consensus_queue.cc:260] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:49.625456 18280 leader_election.cc:341] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.627774 18434 ts_tablet_manager.cc:1398] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20240502 01:49:49.628664 18434 tablet_bootstrap.cc:492] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.631036 18435 ts_tablet_manager.cc:1398] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 01:49:49.632030 18435 tablet_bootstrap.cc:492] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.633348 18392 raft_consensus.cc:2454] T 81a9d50e756d4871a8789ea75c671cca P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.635908 18494 consensus_queue.cc:1035] T af7889fa2a16487d8b767d51bb21abf2 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.637120 18435 tablet_bootstrap.cc:654] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.638032 18280 leader_election.cc:302] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.638716 18391 raft_consensus.cc:2454] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:49.639115 18513 raft_consensus.cc:2781] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.639575 18513 raft_consensus.cc:686] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:49.639773 18203 leader_election.cc:302] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.640923 18547 raft_consensus.cc:2781] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.640591 18513 consensus_queue.cc:237] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.641402 18547 raft_consensus.cc:686] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:49.642372 18433 tablet_bootstrap.cc:654] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.642154 18547 consensus_queue.cc:237] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.652184 18515 raft_consensus.cc:482] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.652730 18515 raft_consensus.cc:504] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.653391 18434 tablet_bootstrap.cc:654] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.654793 18515 leader_election.cc:288] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.656435 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.656733 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.657318 18391 raft_consensus.cc:2454] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
W20240502 01:49:49.657850 18280 leader_election.cc:341] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.658531 18280 leader_election.cc:302] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.659559 18475 raft_consensus.cc:2781] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.659946 18475 raft_consensus.cc:482] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.660218 18475 raft_consensus.cc:3037] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.665961 18475 raft_consensus.cc:504] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.669632 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.670251 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.671241 18391 raft_consensus.cc:3037] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.670553 18481 consensus_queue.cc:1035] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:49.670768 18475 leader_election.cc:288] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:49.671202 18280 leader_election.cc:341] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.670598 18435 tablet_bootstrap.cc:492] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.659533 18433 tablet_bootstrap.cc:492] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.676378 18435 ts_tablet_manager.cc:1367] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.000s
I20240502 01:49:49.676519 18433 ts_tablet_manager.cc:1367] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.000s
I20240502 01:49:49.679117 18130 catalog_manager.cc:5484] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.679905 18435 raft_consensus.cc:348] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.679905 18433 raft_consensus.cc:348] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.681560 18433 raft_consensus.cc:374] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.681869 18433 raft_consensus.cc:729] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.682279 18391 raft_consensus.cc:2454] T 4e46b9ec3add4df0b2026081227bbcd2 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.682672 18433 consensus_queue.cc:260] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.683483 18280 leader_election.cc:302] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.684559 18513 raft_consensus.cc:2781] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.681254 18435 raft_consensus.cc:374] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.685274 18433 ts_tablet_manager.cc:1398] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:49:49.685310 18513 raft_consensus.cc:686] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:49.686308 18513 consensus_queue.cc:237] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.689174 18435 raft_consensus.cc:729] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.689819 18435 consensus_queue.cc:260] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.691808 18492 consensus_queue.cc:1035] T 231a18b76f334dc281187e58de8d90e4 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.695171 18433 tablet_bootstrap.cc:492] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.696108 18434 tablet_bootstrap.cc:492] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.696552 18434 ts_tablet_manager.cc:1367] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.001s
I20240502 01:49:49.699205 18435 ts_tablet_manager.cc:1398] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20240502 01:49:49.700033 18435 tablet_bootstrap.cc:492] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.708714 18433 tablet_bootstrap.cc:654] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.708720 18494 consensus_queue.cc:1035] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 01:49:49.723889 18132 catalog_manager.cc:5484] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.725096 18434 raft_consensus.cc:348] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.725847 18434 raft_consensus.cc:374] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.726284 18434 raft_consensus.cc:729] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.726208 18132 catalog_manager.cc:5484] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.726550 18133 catalog_manager.cc:5484] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.727356 18434 consensus_queue.cc:260] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.728140 18547 raft_consensus.cc:482] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.728571 18547 raft_consensus.cc:504] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.730001 18434 ts_tablet_manager.cc:1398] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20240502 01:49:49.730924 18434 tablet_bootstrap.cc:492] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.731132 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.731941 18315 raft_consensus.cc:2379] T ffa5663e2ff0414b948af8d5967266cc P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:49.734917 18435 tablet_bootstrap.cc:654] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.736400 18547 leader_election.cc:288] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.737008 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa5663e2ff0414b948af8d5967266cc" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.738229 18203 leader_election.cc:302] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:49.739406 18547 raft_consensus.cc:3037] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.746790 18547 raft_consensus.cc:2726] T ffa5663e2ff0414b948af8d5967266cc P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:49:49.747663 18356 consensus_peers.cc:485] T cf23c8fedfe4430bbb092f9e927d55e5 P d6698cd7a2d740368c0c89233e16f04c -> Peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Couldn't send request to peer 26c26643de084d6b8acaff73fa67f794. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:49.757994 18438 raft_consensus.cc:482] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.758527 18438 raft_consensus.cc:504] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.760134 18438 leader_election.cc:288] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:49.760749 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c58e38def504b34b36b61873cc4d5b4" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:49.761883 18356 leader_election.cc:341] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.762413 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c58e38def504b34b36b61873cc4d5b4" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:49.763465 18355 leader_election.cc:341] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.763796 18355 leader_election.cc:302] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:49.764464 18438 raft_consensus.cc:2726] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.772933 18434 tablet_bootstrap.cc:654] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.775163 18435 tablet_bootstrap.cc:492] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.775514 18435 ts_tablet_manager.cc:1367] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.001s
I20240502 01:49:49.777637 18433 tablet_bootstrap.cc:492] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.777969 18434 tablet_bootstrap.cc:492] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.778033 18433 ts_tablet_manager.cc:1367] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.004s
I20240502 01:49:49.777690 18435 raft_consensus.cc:348] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.778434 18434 ts_tablet_manager.cc:1367] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.002s
I20240502 01:49:49.778430 18435 raft_consensus.cc:374] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.780220 18435 raft_consensus.cc:729] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.780961 18435 consensus_queue.cc:260] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.781284 18434 raft_consensus.cc:348] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.781976 18434 raft_consensus.cc:374] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.782233 18434 raft_consensus.cc:729] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.782814 18434 consensus_queue.cc:260] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.780426 18433 raft_consensus.cc:348] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.783627 18433 raft_consensus.cc:374] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.783917 18433 raft_consensus.cc:729] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.784804 18434 ts_tablet_manager.cc:1398] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 01:49:49.784634 18433 consensus_queue.cc:260] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.786321 18435 ts_tablet_manager.cc:1398] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240502 01:49:49.786823 18433 ts_tablet_manager.cc:1398] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:49:49.787153 18435 tablet_bootstrap.cc:492] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.789520 18434 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.790679 18433 tablet_bootstrap.cc:492] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.794808 18434 tablet_bootstrap.cc:654] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.796429 18433 tablet_bootstrap.cc:654] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.806857 18513 raft_consensus.cc:482] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.807368 18513 raft_consensus.cc:504] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.809298 18513 leader_election.cc:288] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.810915 18435 tablet_bootstrap.cc:654] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.811710 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5d4f10e3a24f8e854fda3cbd4b0a0d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.812201 18433 tablet_bootstrap.cc:492] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.812623 18433 ts_tablet_manager.cc:1367] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
W20240502 01:49:49.813383 18280 leader_election.cc:341] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.814673 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5d4f10e3a24f8e854fda3cbd4b0a0d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.815344 18392 raft_consensus.cc:2454] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:49.815128 18433 raft_consensus.cc:348] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.815958 18433 raft_consensus.cc:374] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.816243 18433 raft_consensus.cc:729] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.817112 18434 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.816929 18433 consensus_queue.cc:260] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.817579 18434 ts_tablet_manager.cc:1367] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.016s
I20240502 01:49:49.819178 18433 ts_tablet_manager.cc:1398] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:49:49.819502 18280 leader_election.cc:302] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.820075 18433 tablet_bootstrap.cc:492] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.820452 18513 raft_consensus.cc:2781] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.820185 18434 raft_consensus.cc:348] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.820814 18513 raft_consensus.cc:482] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.820948 18434 raft_consensus.cc:374] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.822582 18513 raft_consensus.cc:3037] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.825686 18433 tablet_bootstrap.cc:654] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.829670 18513 raft_consensus.cc:504] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.831746 18434 raft_consensus.cc:729] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.832392 18434 consensus_queue.cc:260] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.832446 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5d4f10e3a24f8e854fda3cbd4b0a0d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:49.833354 18513 leader_election.cc:288] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:49.833900 18280 leader_election.cc:341] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.834345 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5d4f10e3a24f8e854fda3cbd4b0a0d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.834914 18392 raft_consensus.cc:3037] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.835654 18435 tablet_bootstrap.cc:492] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.836122 18435 ts_tablet_manager.cc:1367] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.019s
I20240502 01:49:49.843225 18433 tablet_bootstrap.cc:492] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.843602 18434 ts_tablet_manager.cc:1398] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.026s user 0.009s sys 0.003s
I20240502 01:49:49.843695 18433 ts_tablet_manager.cc:1367] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.007s
I20240502 01:49:49.844179 18547 raft_consensus.cc:482] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.844565 18434 tablet_bootstrap.cc:492] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.844666 18435 raft_consensus.cc:348] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.845373 18435 raft_consensus.cc:374] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.845633 18435 raft_consensus.cc:729] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.846210 18433 raft_consensus.cc:348] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.846940 18433 raft_consensus.cc:374] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.846210 18435 consensus_queue.cc:260] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.848932 18435 ts_tablet_manager.cc:1398] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:49:49.849557 18392 raft_consensus.cc:2454] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.849933 18435 tablet_bootstrap.cc:492] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.850037 18434 tablet_bootstrap.cc:654] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.850601 18280 leader_election.cc:302] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.851370 18513 raft_consensus.cc:2781] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.847194 18433 raft_consensus.cc:729] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.851816 18513 raft_consensus.cc:686] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:49.852699 18513 consensus_queue.cc:237] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.844630 18547 raft_consensus.cc:504] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.855160 18435 tablet_bootstrap.cc:654] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.855253 18547 leader_election.cc:288] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.857034 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.857966 18433 consensus_queue.cc:260] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.858726 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:49.860314 18433 ts_tablet_manager.cc:1398] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
W20240502 01:49:49.860857 18203 leader_election.cc:341] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.861783 18433 tablet_bootstrap.cc:492] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
W20240502 01:49:49.862305 18201 leader_election.cc:341] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.864219 18201 leader_election.cc:302] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:49.871106 18433 tablet_bootstrap.cc:654] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.871773 18547 raft_consensus.cc:2726] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:49.871768 18132 catalog_manager.cc:5484] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:49.877053 18434 tablet_bootstrap.cc:492] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.879842 18434 ts_tablet_manager.cc:1367] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.005s
I20240502 01:49:49.882959 18435 tablet_bootstrap.cc:492] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.882880 18434 raft_consensus.cc:348] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.883451 18435 ts_tablet_manager.cc:1367] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.005s
I20240502 01:49:49.883611 18434 raft_consensus.cc:374] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.883951 18434 raft_consensus.cc:729] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.884552 18434 consensus_queue.cc:260] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.885915 18435 raft_consensus.cc:348] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.886593 18435 raft_consensus.cc:374] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.886627 18434 ts_tablet_manager.cc:1398] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:49:49.886958 18435 raft_consensus.cc:729] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.887976 18434 tablet_bootstrap.cc:492] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.887672 18435 consensus_queue.cc:260] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.890780 18438 consensus_queue.cc:1035] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.893702 18434 tablet_bootstrap.cc:654] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.898734 18435 ts_tablet_manager.cc:1398] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:49:49.899780 18435 tablet_bootstrap.cc:492] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:49.905354 18435 tablet_bootstrap.cc:654] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.906814 18515 consensus_queue.cc:1035] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:49.910238 18434 tablet_bootstrap.cc:492] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:49.910692 18434 ts_tablet_manager.cc:1367] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.005s
I20240502 01:49:49.917152 18433 tablet_bootstrap.cc:492] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:49.917632 18433 ts_tablet_manager.cc:1367] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.030s
I20240502 01:49:49.918184 18441 consensus_queue.cc:1035] T ffa5663e2ff0414b948af8d5967266cc P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.920050 18433 raft_consensus.cc:348] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.920758 18433 raft_consensus.cc:374] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.921015 18433 raft_consensus.cc:729] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.921705 18433 consensus_queue.cc:260] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.923553 18433 ts_tablet_manager.cc:1398] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 01:49:49.925045 18433 tablet_bootstrap.cc:492] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:49.929888 18515 consensus_queue.cc:1035] T 7943deee0b2f49cf8a391e59d9676aa1 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:49.930706 18433 tablet_bootstrap.cc:654] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.932322 18515 raft_consensus.cc:482] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:49.932767 18515 raft_consensus.cc:504] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.936098 18515 leader_election.cc:288] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.936371 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:49.937045 18391 raft_consensus.cc:2454] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:49.938206 18280 leader_election.cc:302] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:49.939004 18515 raft_consensus.cc:2781] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:49.939350 18515 raft_consensus.cc:482] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:49.939672 18515 raft_consensus.cc:3037] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.956748 18515 raft_consensus.cc:504] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.958894 18515 leader_election.cc:288] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:49.960428 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:49.961576 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:49.962236 18391 raft_consensus.cc:3037] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:49.963850 18434 raft_consensus.cc:348] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.964612 18434 raft_consensus.cc:374] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.969933 18434 raft_consensus.cc:729] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
W20240502 01:49:49.970566 18280 leader_election.cc:341] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.970522 18434 consensus_queue.cc:260] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.972741 18434 ts_tablet_manager.cc:1398] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20240502 01:49:49.973620 18434 tablet_bootstrap.cc:492] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:49.974160 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
W20240502 01:49:49.977366 18280 leader_election.cc:341] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:49.978900 18434 tablet_bootstrap.cc:654] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:49.979841 18391 raft_consensus.cc:2454] T a7d24d3149d14b458dd363496c9d7da9 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:49.983947 18435 tablet_bootstrap.cc:492] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:49.984406 18435 ts_tablet_manager.cc:1367] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.085s user 0.014s sys 0.027s
I20240502 01:49:49.984644 18280 leader_election.cc:302] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:49.985466 18515 raft_consensus.cc:2781] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:49.985971 18515 raft_consensus.cc:686] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:49.986824 18515 consensus_queue.cc:237] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.987294 18435 raft_consensus.cc:348] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.988039 18435 raft_consensus.cc:374] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:49.988318 18435 raft_consensus.cc:729] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:49.988982 18435 consensus_queue.cc:260] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:49.996820 18435 ts_tablet_manager.cc:1398] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240502 01:49:49.997740 18435 tablet_bootstrap.cc:492] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:50.002857 18435 tablet_bootstrap.cc:654] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.006688 18433 tablet_bootstrap.cc:492] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:50.007231 18433 ts_tablet_manager.cc:1367] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.027s
I20240502 01:49:50.009654 18433 raft_consensus.cc:348] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.010422 18433 raft_consensus.cc:374] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.010730 18433 raft_consensus.cc:729] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.010854 18547 raft_consensus.cc:482] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.011358 18547 raft_consensus.cc:504] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.011540 18433 consensus_queue.cc:260] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.012161 18493 raft_consensus.cc:482] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.012629 18493 raft_consensus.cc:504] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.013360 18547 leader_election.cc:288] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.014040 18433 ts_tablet_manager.cc:1398] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 01:49:50.014463 18442 raft_consensus.cc:482] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.015115 18433 tablet_bootstrap.cc:492] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:50.014905 18442 raft_consensus.cc:504] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.016526 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4a55e94ca4287aa230e10d51e21ed" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.016754 18442 leader_election.cc:288] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.018041 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28c47f80264b0fa5cc15992dfd0f8d" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.018307 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4a55e94ca4287aa230e10d51e21ed" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.018707 18314 raft_consensus.cc:2454] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:50.018885 18493 leader_election.cc:288] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:50.020153 18203 leader_election.cc:341] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.020265 18355 leader_election.cc:302] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.021765 18433 tablet_bootstrap.cc:654] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.022210 18441 raft_consensus.cc:2781] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.022601 18441 raft_consensus.cc:482] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.022984 18441 raft_consensus.cc:3037] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.022064 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28c47f80264b0fa5cc15992dfd0f8d" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.028260 18236 raft_consensus.cc:2454] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:50.028769 18435 tablet_bootstrap.cc:492] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:50.029220 18435 ts_tablet_manager.cc:1367] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.012s
I20240502 01:49:50.031543 18435 raft_consensus.cc:348] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.035239 18435 raft_consensus.cc:374] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.035548 18435 raft_consensus.cc:729] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.036151 18435 consensus_queue.cc:260] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.035272 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:50.033690 18201 leader_election.cc:341] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.037828 18201 leader_election.cc:302] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:50.038997 18547 raft_consensus.cc:2726] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:50.039258 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3116ed5b5254b4d926710cf1e5ed93a" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.039630 18435 ts_tablet_manager.cc:1398] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 01:49:50.040074 18391 raft_consensus.cc:2379] T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:50.036579 18130 catalog_manager.cc:5484] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:50.040544 18435 tablet_bootstrap.cc:492] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:50.044087 18442 consensus_queue.cc:1035] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.047582 18433 tablet_bootstrap.cc:492] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:50.048115 18433 ts_tablet_manager.cc:1367] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.013s
I20240502 01:49:50.050421 18433 raft_consensus.cc:348] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.051141 18433 raft_consensus.cc:374] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.051443 18433 raft_consensus.cc:729] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.044180 18280 leader_election.cc:302] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:50.052385 18515 raft_consensus.cc:3037] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.052203 18433 consensus_queue.cc:260] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.054632 18433 ts_tablet_manager.cc:1398] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:49:50.055546 18433 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:50.055565 18435 tablet_bootstrap.cc:654] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.061686 18441 raft_consensus.cc:504] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.064527 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28c47f80264b0fa5cc15992dfd0f8d" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.065183 18236 raft_consensus.cc:3037] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.069859 18433 tablet_bootstrap.cc:654] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.071218 18435 tablet_bootstrap.cc:492] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:50.071720 18435 ts_tablet_manager.cc:1367] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.002s
I20240502 01:49:50.071691 18236 raft_consensus.cc:2454] T ee28c47f80264b0fa5cc15992dfd0f8d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:50.074496 18435 raft_consensus.cc:348] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.075162 18435 raft_consensus.cc:374] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.075428 18435 raft_consensus.cc:729] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.076097 18435 consensus_queue.cc:260] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.078265 18435 ts_tablet_manager.cc:1398] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240502 01:49:50.079102 18435 tablet_bootstrap.cc:492] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
W20240502 01:49:50.083564 18201 consensus_peers.cc:485] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:50.084731 18515 raft_consensus.cc:2726] T e3116ed5b5254b4d926710cf1e5ed93a P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:50.085389 18441 leader_election.cc:288] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.086416 18356 leader_election.cc:302] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.087034 18435 tablet_bootstrap.cc:654] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.087257 18441 raft_consensus.cc:2781] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.088099 18441 raft_consensus.cc:686] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:50.088904 18441 consensus_queue.cc:237] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.084353 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28c47f80264b0fa5cc15992dfd0f8d" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:50.090199 18315 raft_consensus.cc:3037] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.100721 18315 raft_consensus.cc:2454] T ee28c47f80264b0fa5cc15992dfd0f8d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:50.092417 18515 raft_consensus.cc:482] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.106175 18547 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.110078 18239 raft_consensus.cc:3037] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.114614 18515 raft_consensus.cc:504] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.116664 18515 leader_election.cc:288] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.116909 18547 consensus_queue.cc:1035] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.119028 18435 tablet_bootstrap.cc:492] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:50.119673 18435 ts_tablet_manager.cc:1367] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.005s
I20240502 01:49:50.118963 18236 raft_consensus.cc:3037] T 81a9d50e756d4871a8789ea75c671cca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.122400 18435 raft_consensus.cc:348] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.122790 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b741471cb14483b215e1db8dc72dae" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.123328 18435 raft_consensus.cc:374] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.123705 18435 raft_consensus.cc:729] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.124943 18435 consensus_queue.cc:260] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.127656 18435 ts_tablet_manager.cc:1398] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240502 01:49:50.128144 18238 raft_consensus.cc:2454] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:50.131088 18433 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:50.131522 18433 ts_tablet_manager.cc:1367] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.022s
I20240502 01:49:50.133704 18515 raft_consensus.cc:482] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.133965 18280 leader_election.cc:302] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:50.133870 18433 raft_consensus.cc:348] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.134635 18433 raft_consensus.cc:374] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.134271 18515 raft_consensus.cc:504] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.134938 18433 raft_consensus.cc:729] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.134824 18442 raft_consensus.cc:2781] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.136500 18442 raft_consensus.cc:482] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.136554 18515 leader_election.cc:288] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.136933 18442 raft_consensus.cc:3037] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.137686 18435 tablet_bootstrap.cc:492] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:50.141434 18433 consensus_queue.cc:260] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.143532 18433 ts_tablet_manager.cc:1398] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240502 01:49:50.144168 18238 raft_consensus.cc:3037] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.144382 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5090d855c6864a3d904fe1141d503d9d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.145002 18392 raft_consensus.cc:2454] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:50.146338 18433 tablet_bootstrap.cc:492] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:50.124200 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b741471cb14483b215e1db8dc72dae" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.147181 18391 raft_consensus.cc:2454] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:50.148051 18513 consensus_queue.cc:1035] T 81a9d50e756d4871a8789ea75c671cca P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.150503 18280 leader_election.cc:302] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.151535 18433 tablet_bootstrap.cc:654] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.153585 18435 tablet_bootstrap.cc:654] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.154448 18434 tablet_bootstrap.cc:492] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:50.154654 18513 raft_consensus.cc:2781] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.155061 18513 raft_consensus.cc:482] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.155445 18513 raft_consensus.cc:3037] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.154873 18434 ts_tablet_manager.cc:1367] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.181s user 0.021s sys 0.021s
I20240502 01:49:50.157032 18515 consensus_queue.cc:1035] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.158352 18129 catalog_manager.cc:5484] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:50.160086 18434 raft_consensus.cc:348] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.160625 18442 raft_consensus.cc:504] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.155117 18441 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.162029 18441 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.163699 18442 leader_election.cc:288] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.169695 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.226508 18451 raft_consensus.cc:482] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.165479 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.293658 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5090d855c6864a3d904fe1141d503d9d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.166198 18441 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.294487 18239 raft_consensus.cc:2454] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:50.166885 18513 raft_consensus.cc:504] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.295447 18451 raft_consensus.cc:504] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.296629 18513 leader_election.cc:288] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.160754 18434 raft_consensus.cc:374] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.297207 18438 raft_consensus.cc:482] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.297390 18434 raft_consensus.cc:729] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.302619 18434 consensus_queue.cc:260] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.291126 18547 raft_consensus.cc:482] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.164660 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b741471cb14483b215e1db8dc72dae" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:50.225776 18515 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.315639 18434 ts_tablet_manager.cc:1398] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.159s user 0.018s sys 0.015s
I20240502 01:49:50.316486 18434 tablet_bootstrap.cc:492] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:50.308722 18391 raft_consensus.cc:3037] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.302081 18438 raft_consensus.cc:504] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.306758 18547 raft_consensus.cc:504] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.332540 18391 raft_consensus.cc:2454] T 89b741471cb14483b215e1db8dc72dae P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:50.333447 18547 leader_election.cc:288] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.322700 18433 tablet_bootstrap.cc:492] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:50.336163 18433 ts_tablet_manager.cc:1367] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.190s user 0.015s sys 0.014s
I20240502 01:49:50.338582 18433 raft_consensus.cc:348] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.339254 18433 raft_consensus.cc:374] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.339499 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5d4f10e3a24f8e854fda3cbd4b0a0d" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.333451 18438 leader_election.cc:288] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.342123 18547 consensus_queue.cc:1035] T e3116ed5b5254b4d926710cf1e5ed93a P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.340549 18438 raft_consensus.cc:482] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.343722 18438 raft_consensus.cc:504] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.345631 18438 leader_election.cc:288] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.346101 18438 raft_consensus.cc:482] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.346513 18438 raft_consensus.cc:504] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.309319 18515 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.325243 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.329497 18475 consensus_queue.cc:1035] T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.351202 18315 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:50.314277 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b741471cb14483b215e1db8dc72dae" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.353886 18438 leader_election.cc:288] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.352835 18515 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.339524 18433 raft_consensus.cc:729] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.354185 18239 raft_consensus.cc:3037] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.376327 18451 leader_election.cc:288] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.441854 18433 consensus_queue.cc:260] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.443655 18435 tablet_bootstrap.cc:492] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:50.444227 18435 ts_tablet_manager.cc:1367] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.307s user 0.025s sys 0.015s
I20240502 01:49:50.453382 18433 ts_tablet_manager.cc:1398] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.117s user 0.015s sys 0.008s
I20240502 01:49:50.454064 18433 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:50.450690 18604 raft_consensus.cc:482] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.452004 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.457063 18237 raft_consensus.cc:2454] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:50.458703 18604 raft_consensus.cc:504] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.460777 18604 leader_election.cc:288] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.465703 18434 tablet_bootstrap.cc:654] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:50.446138 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 111601us
W20240502 01:49:50.472498 18600 log.cc:924] Time spent T e3116ed5b5254b4d926710cf1e5ed93a P d6698cd7a2d740368c0c89233e16f04c: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
I20240502 01:49:50.473759 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5090d855c6864a3d904fe1141d503d9d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:50.474439 18392 raft_consensus.cc:3037] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.479063 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.479460 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5090d855c6864a3d904fe1141d503d9d" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.479806 18391 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:50.480154 18237 raft_consensus.cc:3037] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.476590 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4a55e94ca4287aa230e10d51e21ed" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.498193 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.501372 18233 raft_consensus.cc:3037] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.494890 18236 raft_consensus.cc:2454] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:50.504603 18280 leader_election.cc:302] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
W20240502 01:49:50.502947 18356 leader_election.cc:341] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.521495 18435 raft_consensus.cc:348] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.524564 18435 raft_consensus.cc:374] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.524838 18435 raft_consensus.cc:729] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.514039 18515 raft_consensus.cc:2781] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.524976 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4a55e94ca4287aa230e10d51e21ed" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.526348 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.527012 18234 raft_consensus.cc:2454] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:50.529073 18435 consensus_queue.cc:260] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.537993 18356 leader_election.cc:302] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.515636 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.542029 18280 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.541050 18356 leader_election.cc:302] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.563817 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c58e38def504b34b36b61873cc4d5b4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.565441 18442 consensus_queue.cc:1035] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:50.527541 18515 raft_consensus.cc:686] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
W20240502 01:49:50.572300 18201 leader_election.cc:341] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.577951 18493 raft_consensus.cc:2781] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.578621 18493 raft_consensus.cc:482] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.577994 18442 raft_consensus.cc:482] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.579011 18493 raft_consensus.cc:3037] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.579355 18442 raft_consensus.cc:504] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.549858 18513 raft_consensus.cc:2781] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.583562 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c58e38def504b34b36b61873cc4d5b4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.561760 18611 raft_consensus.cc:2781] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.585234 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e46b9ec3add4df0b2026081227bbcd2" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.585783 18481 raft_consensus.cc:482] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6698cd7a2d740368c0c89233e16f04c)
I20240502 01:49:50.597491 18611 raft_consensus.cc:482] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.597916 18611 raft_consensus.cc:3037] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.539641 18435 ts_tablet_manager.cc:1398] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.095s user 0.007s sys 0.002s
I20240502 01:49:50.605141 18435 tablet_bootstrap.cc:492] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:50.584317 18387 raft_consensus.cc:2454] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:50.541574 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.584820 18513 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.609269 18513 raft_consensus.cc:3037] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.605974 18356 leader_election.cc:302] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.611361 18433 tablet_bootstrap.cc:654] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:50.579170 18280 leader_election.cc:341] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.597014 18481 raft_consensus.cc:504] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.615242 18494 raft_consensus.cc:482] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.615763 18392 raft_consensus.cc:2454] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:50.587195 18515 consensus_queue.cc:237] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.621268 18494 raft_consensus.cc:504] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.624131 18355 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:50.625403 18494 leader_election.cc:288] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.625486 18239 raft_consensus.cc:2454] T 89b741471cb14483b215e1db8dc72dae P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:50.623704 18614 raft_consensus.cc:2781] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 01:49:50.622227 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:50.612782 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5d4f10e3a24f8e854fda3cbd4b0a0d" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.637456 18614 raft_consensus.cc:482] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.637910 18614 raft_consensus.cc:3037] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.638168 18494 raft_consensus.cc:2781] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.653048 18434 tablet_bootstrap.cc:492] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:50.653501 18434 ts_tablet_manager.cc:1367] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.337s user 0.021s sys 0.017s
I20240502 01:49:50.650261 18494 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:50.650954 18390 raft_consensus.cc:2379] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:50.649176 18481 leader_election.cc:288] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.649756 18442 leader_election.cc:288] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.656183 18203 leader_election.cc:302] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:50.661360 18434 raft_consensus.cc:348] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.727213 18434 raft_consensus.cc:374] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.727468 18434 raft_consensus.cc:729] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.656230 18494 raft_consensus.cc:3037] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:50.654901 18355 leader_election.cc:341] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.729693 18434 consensus_queue.cc:260] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.735180 18434 ts_tablet_manager.cc:1398] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.081s user 0.018s sys 0.012s
I20240502 01:49:50.736568 18434 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
W20240502 01:49:50.737032 18615 log.cc:924] Time spent T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20240502 01:49:50.740275 18608 raft_consensus.cc:2781] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:50.740732 18608 raft_consensus.cc:482] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 01:49:50.743719 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 90643us
W20240502 01:49:50.745815 18355 leader_election.cc:341] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.735855 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7dd74b7ba34a7dab5ada581af71795" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.743746 18608 raft_consensus.cc:3037] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:50.750582 18203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 94502.1us
I20240502 01:49:50.769340 18435 tablet_bootstrap.cc:654] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.770815 18434 tablet_bootstrap.cc:654] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:50.775671 18355 leader_election.cc:341] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.784557 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf23c8fedfe4430bbb092f9e927d55e5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.782430 18203 leader_election.cc:302] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:50.789896 18513 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:50.787021 18356 leader_election.cc:341] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.792732 18513 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.791519 18608 raft_consensus.cc:504] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.793651 18605 consensus_queue.cc:1035] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:50.786777 18237 raft_consensus.cc:2454] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:50.796116 18604 raft_consensus.cc:2726] T 4e46b9ec3add4df0b2026081227bbcd2 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:49:50.791625 18493 raft_consensus.cc:504] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.808801 18433 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:50.809289 18433 ts_tablet_manager.cc:1367] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.355s user 0.015s sys 0.031s
I20240502 01:49:50.811250 18280 leader_election.cc:302] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.811800 18493 leader_election.cc:288] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.812474 18203 leader_election.cc:302] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:50.813130 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7dd74b7ba34a7dab5ada581af71795" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.813364 18604 raft_consensus.cc:2726] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:50.814162 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c58e38def504b34b36b61873cc4d5b4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:50.814764 18313 raft_consensus.cc:3037] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.814947 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf23c8fedfe4430bbb092f9e927d55e5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.816262 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4a55e94ca4287aa230e10d51e21ed" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.816890 18238 raft_consensus.cc:3037] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.807183 18611 raft_consensus.cc:504] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.826398 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:50.806679 18608 leader_election.cc:288] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.828537 18433 raft_consensus.cc:348] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.829375 18433 raft_consensus.cc:374] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.829643 18433 raft_consensus.cc:729] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.834239 18433 consensus_queue.cc:260] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.837312 18614 raft_consensus.cc:504] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.839497 18614 leader_election.cc:288] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.841202 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.841917 18233 raft_consensus.cc:3037] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.799707 18129 catalog_manager.cc:5484] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
W20240502 01:49:50.844714 18280 consensus_peers.cc:485] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Couldn't send request to peer 26c26643de084d6b8acaff73fa67f794. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:50.846899 18611 leader_election.cc:288] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.879369 18623 consensus_queue.cc:1035] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 01:49:50.843649 18238 raft_consensus.cc:2454] T acc4a55e94ca4287aa230e10d51e21ed P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:50.886876 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:50.887223 18433 ts_tablet_manager.cc:1398] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.078s user 0.003s sys 0.003s
W20240502 01:49:50.882586 18625 log.cc:924] Time spent T 4e46b9ec3add4df0b2026081227bbcd2 P 98f3bf374d49481cb6ebe0672e5b81dc: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20240502 01:49:50.877892 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.892503 18356 leader_election.cc:302] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.893121 18232 raft_consensus.cc:3037] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.893348 18614 raft_consensus.cc:2781] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.893815 18614 raft_consensus.cc:686] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:50.894666 18433 tablet_bootstrap.cc:492] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:50.897491 18494 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.898653 18392 raft_consensus.cc:2379] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:50.899413 18494 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:50.902873 18435 tablet_bootstrap.cc:492] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:50.903215 18435 ts_tablet_manager.cc:1367] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.298s user 0.022s sys 0.016s
I20240502 01:49:50.891603 18313 raft_consensus.cc:2454] T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:50.907841 18620 raft_consensus.cc:2781] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.908425 18620 raft_consensus.cc:686] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:50.909237 18620 consensus_queue.cc:237] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.910601 18435 raft_consensus.cc:348] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.911393 18435 raft_consensus.cc:374] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:50.911679 18435 raft_consensus.cc:729] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:50.911880 18201 leader_election.cc:302] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:50.912048 18614 consensus_queue.cc:237] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.913259 18604 raft_consensus.cc:482] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:50.913703 18604 raft_consensus.cc:504] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.913826 18481 raft_consensus.cc:2781] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.915578 18481 raft_consensus.cc:686] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:50.916420 18481 consensus_queue.cc:237] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.911473 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c58e38def504b34b36b61873cc4d5b4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:50.920773 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.912420 18435 consensus_queue.cc:260] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.921487 18233 raft_consensus.cc:2454] T 6231687fc05f4510bf8b98b53dca0345 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:50.915661 18604 leader_election.cc:288] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:50.929363 18387 raft_consensus.cc:3037] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:50.936300 18435 ts_tablet_manager.cc:1398] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20240502 01:49:50.937522 18435 tablet_bootstrap.cc:492] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
W20240502 01:49:50.928274 18355 leader_election.cc:341] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.950141 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:50.951233 18356 leader_election.cc:302] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.951900 18433 tablet_bootstrap.cc:654] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:50.952163 18611 raft_consensus.cc:2781] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.952495 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:50.953755 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:50.952673 18611 raft_consensus.cc:686] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
W20240502 01:49:50.956480 18356 leader_election.cc:341] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.956727 18611 consensus_queue.cc:237] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:50.959914 18201 leader_election.cc:341] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.953334 18355 leader_election.cc:302] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:50.963120 18493 raft_consensus.cc:2726] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:49:50.963979 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 44356.2us
I20240502 01:49:50.965006 18232 raft_consensus.cc:2454] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:50.970448 18356 leader_election.cc:302] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:50.970361 18131 catalog_manager.cc:5484] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:50.971354 18493 raft_consensus.cc:2781] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:50.974097 18493 raft_consensus.cc:686] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:50.980046 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc4a55e94ca4287aa230e10d51e21ed" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:50.974958 18493 consensus_queue.cc:237] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:50.979908 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:50.986656 18280 leader_election.cc:341] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.986521 18620 consensus_queue.cc:1035] T a7d24d3149d14b458dd363496c9d7da9 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:49:50.987399 18280 leader_election.cc:341] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.986747 18203 leader_election.cc:302] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:50.988685 18481 raft_consensus.cc:2726] T cf23c8fedfe4430bbb092f9e927d55e5 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:49:50.990666 18434 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:50.991130 18434 ts_tablet_manager.cc:1367] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.255s user 0.025s sys 0.015s
W20240502 01:49:50.991387 18203 leader_election.cc:341] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:50.995559 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:51.004215 18435 tablet_bootstrap.cc:654] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.007827 18434 raft_consensus.cc:348] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.008646 18434 raft_consensus.cc:374] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.008921 18434 raft_consensus.cc:729] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.009522 18434 consensus_queue.cc:260] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.021010 18434 ts_tablet_manager.cc:1398] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20240502 01:49:51.022524 18434 tablet_bootstrap.cc:492] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:51.001955 18203 leader_election.cc:302] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.024367 18451 raft_consensus.cc:2726] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.023466 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:51.018105 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:51.026737 18315 raft_consensus.cc:2379] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:51.027119 18280 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.029695 18442 raft_consensus.cc:2726] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.037055 18387 raft_consensus.cc:2454] T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
W20240502 01:49:51.044734 18355 leader_election.cc:341] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:51.046044 18355 leader_election.cc:341] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.048868 18433 tablet_bootstrap.cc:492] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.049404 18433 ts_tablet_manager.cc:1367] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.011s
I20240502 01:49:51.055933 18434 tablet_bootstrap.cc:654] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.055577 18433 raft_consensus.cc:348] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.056311 18433 raft_consensus.cc:374] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.056608 18433 raft_consensus.cc:729] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.057263 18433 consensus_queue.cc:260] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:51.062016 18280 leader_election.cc:341] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.062494 18280 leader_election.cc:302] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.063290 18623 raft_consensus.cc:2726] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.066305 18433 ts_tablet_manager.cc:1398] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20240502 01:49:51.067207 18433 tablet_bootstrap.cc:492] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.071074 18624 consensus_queue.cc:1035] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:51.073944 18441 consensus_queue.cc:1035] T ee28c47f80264b0fa5cc15992dfd0f8d P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:51.075640 18355 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:51.076424 18441 raft_consensus.cc:2726] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240502 01:49:51.077564 18355 leader_election.cc:341] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.080492 18435 tablet_bootstrap.cc:492] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.081728 18435 ts_tablet_manager.cc:1367] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.006s
I20240502 01:49:51.084225 18435 raft_consensus.cc:348] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.089041 18435 raft_consensus.cc:374] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.091405 18435 raft_consensus.cc:729] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.092228 18435 consensus_queue.cc:260] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.085686 18433 tablet_bootstrap.cc:654] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.086918 18434 tablet_bootstrap.cc:492] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:51.096261 18434 ts_tablet_manager.cc:1367] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.074s user 0.017s sys 0.006s
I20240502 01:49:51.097307 18435 ts_tablet_manager.cc:1398] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:49:51.098925 18435 tablet_bootstrap.cc:492] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:51.098670 18434 raft_consensus.cc:348] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.099411 18434 raft_consensus.cc:374] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.099697 18434 raft_consensus.cc:729] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.100337 18434 consensus_queue.cc:260] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.103165 18434 ts_tablet_manager.cc:1398] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:49:51.104053 18434 tablet_bootstrap.cc:492] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:51.104763 18622 consensus_queue.cc:1035] T 89b741471cb14483b215e1db8dc72dae P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:51.106523 18435 tablet_bootstrap.cc:654] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.109167 18434 tablet_bootstrap.cc:654] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.114488 18618 raft_consensus.cc:482] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.115026 18618 raft_consensus.cc:504] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.117342 18618 leader_election.cc:288] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.119241 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e09bfc9fdb46af9dd702b4c831e287" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.111138 18131 catalog_manager.cc:5484] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:51.124006 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5e09bfc9fdb46af9dd702b4c831e287" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.146397 18435 tablet_bootstrap.cc:492] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.146934 18435 ts_tablet_manager.cc:1367] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.014s
I20240502 01:49:51.149365 18435 raft_consensus.cc:348] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.150087 18435 raft_consensus.cc:374] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.150333 18435 raft_consensus.cc:729] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.150792 18280 leader_election.cc:302] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.150868 18435 consensus_queue.cc:260] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.152345 18620 raft_consensus.cc:2726] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.154704 18435 ts_tablet_manager.cc:1398] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240502 01:49:51.157661 18433 tablet_bootstrap.cc:492] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.158111 18433 ts_tablet_manager.cc:1367] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.091s user 0.022s sys 0.008s
I20240502 01:49:51.158181 18614 raft_consensus.cc:482] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.158653 18614 raft_consensus.cc:504] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.159230 18451 raft_consensus.cc:482] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.159729 18451 raft_consensus.cc:504] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.160655 18614 leader_election.cc:288] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.160457 18433 raft_consensus.cc:348] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.161125 18433 raft_consensus.cc:374] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.161406 18433 raft_consensus.cc:729] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.162184 18620 raft_consensus.cc:482] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.162003 18433 consensus_queue.cc:260] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.162454 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.162611 18620 raft_consensus.cc:504] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.164098 18433 ts_tablet_manager.cc:1398] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:49:51.164211 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92fc4beef0b4747bcf45cbabe4bb118" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.164522 18451 leader_election.cc:288] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.155654 18435 tablet_bootstrap.cc:492] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:51.164947 18433 tablet_bootstrap.cc:492] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.166263 18238 raft_consensus.cc:3037] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.181838 18435 tablet_bootstrap.cc:654] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.179319 18133 catalog_manager.cc:5484] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:51.167446 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d24d3149d14b458dd363496c9d7da9" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.183549 18133 catalog_manager.cc:5484] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
W20240502 01:49:51.172268 18356 leader_election.cc:341] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.172932 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.185113 18203 leader_election.cc:302] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.174533 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1b268530a4a38a3a366888b97b663" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.185313 18233 raft_consensus.cc:2379] T 26c1b268530a4a38a3a366888b97b663 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:51.170490 18620 leader_election.cc:288] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.165958 18604 raft_consensus.cc:482] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.173717 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92fc4beef0b4747bcf45cbabe4bb118" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.186429 18604 raft_consensus.cc:504] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:51.187919 18355 leader_election.cc:341] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.188434 18355 leader_election.cc:302] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:51.189955 18280 leader_election.cc:302] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.185305 18433 tablet_bootstrap.cc:654] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.190104 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282fd6d7ad0249509f67bfcbc4c0adf5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.191056 18620 raft_consensus.cc:3037] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:51.191619 18201 leader_election.cc:341] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.188544 18604 leader_election.cc:288] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.192647 18133 catalog_manager.cc:5484] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:51.196012 18614 raft_consensus.cc:2726] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.197626 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282fd6d7ad0249509f67bfcbc4c0adf5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.198277 18433 tablet_bootstrap.cc:492] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.198701 18433 ts_tablet_manager.cc:1367] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.004s
W20240502 01:49:51.198820 18203 leader_election.cc:341] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.199220 18203 leader_election.cc:302] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.199797 18604 raft_consensus.cc:2726] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.201357 18433 raft_consensus.cc:348] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.202127 18433 raft_consensus.cc:374] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.202376 18433 raft_consensus.cc:729] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.203234 18620 raft_consensus.cc:2726] T 26c1b268530a4a38a3a366888b97b663 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.203147 18433 consensus_queue.cc:260] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.204636 18434 tablet_bootstrap.cc:492] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:51.205065 18434 ts_tablet_manager.cc:1367] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.101s user 0.018s sys 0.011s
I20240502 01:49:51.205862 18433 ts_tablet_manager.cc:1398] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:49:51.206558 18433 tablet_bootstrap.cc:492] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.210541 18451 raft_consensus.cc:2726] T a7d24d3149d14b458dd363496c9d7da9 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.211349 18435 tablet_bootstrap.cc:492] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.211681 18435 ts_tablet_manager.cc:1367] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.010s
I20240502 01:49:51.207458 18434 raft_consensus.cc:348] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.212951 18434 raft_consensus.cc:374] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.213268 18434 raft_consensus.cc:729] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.213397 18435 raft_consensus.cc:348] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.213869 18435 raft_consensus.cc:374] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.213886 18434 consensus_queue.cc:260] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.216261 18434 ts_tablet_manager.cc:1398] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:49:51.216382 18433 tablet_bootstrap.cc:654] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.217120 18434 tablet_bootstrap.cc:492] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:51.214099 18435 raft_consensus.cc:729] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.219507 18435 consensus_queue.cc:260] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.221493 18435 ts_tablet_manager.cc:1398] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:49:51.222334 18435 tablet_bootstrap.cc:492] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:51.222730 18434 tablet_bootstrap.cc:654] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.227641 18435 tablet_bootstrap.cc:654] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.277526 18433 tablet_bootstrap.cc:492] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.278101 18433 ts_tablet_manager.cc:1367] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.072s user 0.034s sys 0.029s
I20240502 01:49:51.280637 18433 raft_consensus.cc:348] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.281322 18433 raft_consensus.cc:374] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.281584 18433 raft_consensus.cc:729] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.282174 18433 consensus_queue.cc:260] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.284201 18435 tablet_bootstrap.cc:492] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.284509 18433 ts_tablet_manager.cc:1398] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 01:49:51.284641 18434 tablet_bootstrap.cc:492] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:51.285274 18434 ts_tablet_manager.cc:1367] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.068s user 0.034s sys 0.022s
I20240502 01:49:51.285429 18433 tablet_bootstrap.cc:492] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.284680 18435 ts_tablet_manager.cc:1367] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.063s user 0.037s sys 0.023s
I20240502 01:49:51.287786 18434 raft_consensus.cc:348] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.288190 18435 raft_consensus.cc:348] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.288578 18434 raft_consensus.cc:374] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.288851 18435 raft_consensus.cc:374] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.288874 18434 raft_consensus.cc:729] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.289440 18435 raft_consensus.cc:729] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.291849 18433 tablet_bootstrap.cc:654] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.290066 18434 consensus_queue.cc:260] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.290048 18435 consensus_queue.cc:260] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.293785 18434 ts_tablet_manager.cc:1398] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 01:49:51.294816 18434 tablet_bootstrap.cc:492] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:51.301970 18434 tablet_bootstrap.cc:654] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.305675 18435 ts_tablet_manager.cc:1398] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240502 01:49:51.317507 18435 tablet_bootstrap.cc:492] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:51.323094 18435 tablet_bootstrap.cc:654] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.324498 18434 tablet_bootstrap.cc:492] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:51.325031 18434 ts_tablet_manager.cc:1367] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.004s
I20240502 01:49:51.325130 18433 tablet_bootstrap.cc:492] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.325603 18433 ts_tablet_manager.cc:1367] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.021s
I20240502 01:49:51.329406 18434 raft_consensus.cc:348] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.330276 18434 raft_consensus.cc:374] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.330046 18614 raft_consensus.cc:482] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.330031 18433 raft_consensus.cc:348] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.330796 18434 raft_consensus.cc:729] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.330919 18433 raft_consensus.cc:374] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.331461 18433 raft_consensus.cc:729] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.330916 18614 raft_consensus.cc:504] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.331799 18434 consensus_queue.cc:260] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.333156 18614 leader_election.cc:288] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.334084 18434 ts_tablet_manager.cc:1398] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:49:51.335681 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac68a1674ab4b15a8274884f20a0266" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.336640 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac68a1674ab4b15a8274884f20a0266" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.337281 18238 raft_consensus.cc:2454] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:51.335216 18433 consensus_queue.cc:260] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.337935 18434 tablet_bootstrap.cc:492] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
W20240502 01:49:51.338471 18355 leader_election.cc:341] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.340269 18356 leader_election.cc:302] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:51.345217 18493 raft_consensus.cc:2781] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:51.345817 18434 tablet_bootstrap.cc:654] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.345693 18493 raft_consensus.cc:482] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:51.348906 18433 ts_tablet_manager.cc:1398] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.023s user 0.005s sys 0.003s
I20240502 01:49:51.349226 18493 raft_consensus.cc:3037] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.349884 18433 tablet_bootstrap.cc:492] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.360132 18433 tablet_bootstrap.cc:654] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.361389 18441 consensus_queue.cc:1035] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:51.372844 18620 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:51.373513 18315 raft_consensus.cc:3037] T acc4a55e94ca4287aa230e10d51e21ed P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.376538 18493 raft_consensus.cc:504] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.379577 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac68a1674ab4b15a8274884f20a0266" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:51.380484 18493 leader_election.cc:288] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:51.380789 18355 leader_election.cc:341] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.381096 18435 tablet_bootstrap.cc:492] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.381598 18435 ts_tablet_manager.cc:1367] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.004s
I20240502 01:49:51.384183 18435 raft_consensus.cc:348] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.384471 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac68a1674ab4b15a8274884f20a0266" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:51.384914 18435 raft_consensus.cc:374] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.385097 18232 raft_consensus.cc:3037] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.387063 18434 tablet_bootstrap.cc:492] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:51.387645 18434 ts_tablet_manager.cc:1367] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.009s
I20240502 01:49:51.388073 18620 raft_consensus.cc:482] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.388576 18620 raft_consensus.cc:504] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.390905 18434 raft_consensus.cc:348] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.391417 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7dd74b7ba34a7dab5ada581af71795" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.385214 18435 raft_consensus.cc:729] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.392093 18239 raft_consensus.cc:2454] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:51.392525 18614 consensus_queue.cc:1035] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:51.392377 18435 consensus_queue.cc:260] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.393309 18280 leader_election.cc:302] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:51.394408 18620 leader_election.cc:288] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.394922 18435 ts_tablet_manager.cc:1398] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240502 01:49:51.394188 18618 raft_consensus.cc:2781] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:51.395604 18618 raft_consensus.cc:482] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:51.395987 18618 raft_consensus.cc:3037] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.396761 18435 tablet_bootstrap.cc:492] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:51.397990 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7dd74b7ba34a7dab5ada581af71795" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.398540 18434 raft_consensus.cc:374] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.398636 18388 raft_consensus.cc:2454] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:51.398725 18614 consensus_queue.cc:1035] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:51.399361 18434 raft_consensus.cc:729] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.400257 18434 consensus_queue.cc:260] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.402349 18434 ts_tablet_manager.cc:1398] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20240502 01:49:51.404299 18434 tablet_bootstrap.cc:492] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:51.408389 18618 raft_consensus.cc:504] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.407738 18435 tablet_bootstrap.cc:654] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.391536 18232 raft_consensus.cc:2454] T bac68a1674ab4b15a8274884f20a0266 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:51.417150 18620 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:51.420066 18614 raft_consensus.cc:482] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.420593 18614 raft_consensus.cc:504] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.431627 18614 leader_election.cc:288] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.435155 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.442240 18434 tablet_bootstrap.cc:654] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:51.442117 18355 leader_election.cc:341] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.444869 18433 tablet_bootstrap.cc:492] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.445345 18433 ts_tablet_manager.cc:1367] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.012s
I20240502 01:49:51.442953 18618 leader_election.cc:288] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.442605 18451 consensus_queue.cc:1035] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:51.448977 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7dd74b7ba34a7dab5ada581af71795" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:51.450947 18387 raft_consensus.cc:3037] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.454327 18356 leader_election.cc:302] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:51.455287 18441 raft_consensus.cc:2781] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:51.455984 18620 raft_consensus.cc:482] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.455922 18441 raft_consensus.cc:686] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:51.457170 18441 consensus_queue.cc:237] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.459080 18620 raft_consensus.cc:504] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.461292 18620 leader_election.cc:288] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.461589 18314 raft_consensus.cc:3037] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.462014 18451 consensus_queue.cc:1035] T 8c58e38def504b34b36b61873cc4d5b4 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:51.442729 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7dd74b7ba34a7dab5ada581af71795" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:51.463245 18611 consensus_queue.cc:1035] T acc4a55e94ca4287aa230e10d51e21ed P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:51.464985 18233 raft_consensus.cc:3037] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.479192 18387 raft_consensus.cc:2454] T ba7dd74b7ba34a7dab5ada581af71795 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:51.480386 18435 tablet_bootstrap.cc:492] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.480916 18435 ts_tablet_manager.cc:1367] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.000s
I20240502 01:49:51.481186 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.482369 18280 leader_election.cc:302] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:51.496757 18435 raft_consensus.cc:348] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.497606 18435 raft_consensus.cc:374] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.497870 18435 raft_consensus.cc:729] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.499395 18435 consensus_queue.cc:260] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.490244 18618 raft_consensus.cc:2781] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:51.502096 18433 raft_consensus.cc:348] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.508347 18433 raft_consensus.cc:374] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.508666 18433 raft_consensus.cc:729] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.509378 18433 consensus_queue.cc:260] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.488126 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.510994 18618 raft_consensus.cc:686] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:51.515656 18435 ts_tablet_manager.cc:1398] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20240502 01:49:51.520326 18435 tablet_bootstrap.cc:492] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:51.518729 18618 consensus_queue.cc:237] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.526268 18233 raft_consensus.cc:2454] T ba7dd74b7ba34a7dab5ada581af71795 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:51.529865 18433 ts_tablet_manager.cc:1398] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.084s user 0.007s sys 0.000s
I20240502 01:49:51.532485 18433 tablet_bootstrap.cc:492] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.515992 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6231687fc05f4510bf8b98b53dca0345" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.538805 18313 raft_consensus.cc:3037] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.542876 18623 raft_consensus.cc:482] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.543442 18623 raft_consensus.cc:504] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:51.552233 18356 leader_election.cc:341] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.565829 18356 leader_election.cc:302] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:51.567267 18493 raft_consensus.cc:2726] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.610981 18614 raft_consensus.cc:482] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.611548 18614 raft_consensus.cc:504] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.601114 18493 raft_consensus.cc:482] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 01:49:51.640906 18670 log.cc:924] Time spent T 8c58e38def504b34b36b61873cc4d5b4 P 98f3bf374d49481cb6ebe0672e5b81dc: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20240502 01:49:51.642930 18608 raft_consensus.cc:482] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.651144 18611 raft_consensus.cc:482] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.651131 18608 raft_consensus.cc:504] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.674119 18605 raft_consensus.cc:482] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.674754 18605 raft_consensus.cc:504] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.676179 18451 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.686077 18280 leader_election.cc:302] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
W20240502 01:49:51.748443 18356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.129:38913, user_credentials={real_user=slave}} blocked reactor thread for 210847us
I20240502 01:49:51.748937 18434 tablet_bootstrap.cc:492] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:51.749433 18434 ts_tablet_manager.cc:1367] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.346s user 0.025s sys 0.043s
I20240502 01:49:51.750664 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:51.752279 18203 leader_election.cc:341] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.753077 18434 raft_consensus.cc:348] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.754191 18434 raft_consensus.cc:374] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.754503 18434 raft_consensus.cc:729] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.755228 18434 consensus_queue.cc:260] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.765460 18434 ts_tablet_manager.cc:1398] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240502 01:49:51.766606 18434 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:51.769124 18433 tablet_bootstrap.cc:654] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.651686 18611 raft_consensus.cc:504] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.759649 18608 leader_election.cc:288] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:51.753229 18280 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 67322.4us
I20240502 01:49:51.772151 18618 raft_consensus.cc:2726] T 6231687fc05f4510bf8b98b53dca0345 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.753105 18493 raft_consensus.cc:504] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.756251 18451 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.780457 18605 leader_election.cc:288] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.793049 18614 leader_election.cc:288] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:51.792244 18201 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 54196us
I20240502 01:49:51.795729 18623 leader_election.cc:288] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.814106 18435 tablet_bootstrap.cc:654] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:51.817193 18493 leader_election.cc:288] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.818639 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282fd6d7ad0249509f67bfcbc4c0adf5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.819308 18238 raft_consensus.cc:2454] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
W20240502 01:49:51.820822 18675 log.cc:924] Time spent T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc: Append to log took a long time: real 0.220s user 0.004s sys 0.000s
W20240502 01:49:51.823479 18671 log.cc:924] Time spent T 8c58e38def504b34b36b61873cc4d5b4 P d6698cd7a2d740368c0c89233e16f04c: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240502 01:49:51.827422 18614 consensus_queue.cc:1035] T 26c1b268530a4a38a3a366888b97b663 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240502 01:49:51.839963 18451 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.843549 18451 raft_consensus.cc:482] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.844152 18451 raft_consensus.cc:504] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.847369 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b45ab88cf544fca9ca712cfaa2c601" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.846668 18451 leader_election.cc:288] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:51.849977 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.850790 18391 raft_consensus.cc:2379] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:51.851771 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.860287 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19377bfe6e48e481f6054c06e669b6" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.848524 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.859691 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19377bfe6e48e481f6054c06e669b6" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.862250 18233 raft_consensus.cc:2454] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:51.846639 18356 leader_election.cc:302] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:51.865100 18493 raft_consensus.cc:2781] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 01:49:51.865195 18356 leader_election.cc:341] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.865533 18493 raft_consensus.cc:482] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:51.865875 18493 raft_consensus.cc:3037] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:51.868808 18203 leader_election.cc:341] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:51.870319 18280 leader_election.cc:341] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.878893 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73530ff3121b43478be31af22ba77d09" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:51.879580 18280 leader_election.cc:341] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.880075 18280 leader_election.cc:302] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:51.876163 18611 leader_election.cc:288] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.895392 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282fd6d7ad0249509f67bfcbc4c0adf5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.898217 18433 tablet_bootstrap.cc:492] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:51.898792 18433 ts_tablet_manager.cc:1367] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.367s user 0.034s sys 0.016s
I20240502 01:49:51.901551 18433 raft_consensus.cc:348] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.902268 18433 raft_consensus.cc:374] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:51.911691 18433 raft_consensus.cc:729] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:51.912366 18433 consensus_queue.cc:260] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.913923 18433 ts_tablet_manager.cc:1398] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240502 01:49:51.915088 18433 tablet_bootstrap.cc:492] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:51.907557 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.909597 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b45ab88cf544fca9ca712cfaa2c601" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.885562 18356 leader_election.cc:302] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:51.921928 18614 raft_consensus.cc:2781] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:51.922359 18614 raft_consensus.cc:482] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:51.922727 18614 raft_consensus.cc:3037] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:51.922750 18356 leader_election.cc:341] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:51.925098 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:51.931643 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73530ff3121b43478be31af22ba77d09" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.932263 18678 consensus_queue.cc:1035] T 6231687fc05f4510bf8b98b53dca0345 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:51.936369 18614 raft_consensus.cc:504] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.901623 18682 raft_consensus.cc:2726] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.948809 18682 raft_consensus.cc:482] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:51.949358 18682 raft_consensus.cc:504] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.952350 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:51.954020 18232 raft_consensus.cc:2454] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:51.963754 18280 leader_election.cc:302] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:51.964867 18620 raft_consensus.cc:2781] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:51.965271 18620 raft_consensus.cc:482] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:51.965617 18620 raft_consensus.cc:3037] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:51.976205 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:51.981014 18493 raft_consensus.cc:504] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:51.982686 18435 tablet_bootstrap.cc:492] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:51.983175 18435 ts_tablet_manager.cc:1367] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.463s user 0.025s sys 0.010s
I20240502 01:49:51.985489 18133 catalog_manager.cc:5484] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
W20240502 01:49:51.989745 18355 leader_election.cc:341] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.991443 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.992134 18313 raft_consensus.cc:2454] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:51.993822 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:51.994575 18313 raft_consensus.cc:2379] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
W20240502 01:49:51.995801 18355 leader_election.cc:341] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:51.996766 18355 leader_election.cc:302] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:51.997505 18611 raft_consensus.cc:2726] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:51.997846 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:51.999658 18355 leader_election.cc:341] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:52.001698 18355 leader_election.cc:341] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.002177 18355 leader_election.cc:302] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.002856 18611 raft_consensus.cc:2726] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.003387 18682 leader_election.cc:288] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.007459 18493 leader_election.cc:288] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:51.974704 18620 raft_consensus.cc:504] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.012974 18434 tablet_bootstrap.cc:654] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.015157 18435 raft_consensus.cc:348] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.015728 18435 raft_consensus.cc:374] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.016831 18201 leader_election.cc:302] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.017714 18608 raft_consensus.cc:2781] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.018061 18608 raft_consensus.cc:482] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.018375 18608 raft_consensus.cc:3037] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.021437 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282fd6d7ad0249509f67bfcbc4c0adf5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:52.022082 18238 raft_consensus.cc:3037] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.015976 18435 raft_consensus.cc:729] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.024950 18433 tablet_bootstrap.cc:654] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.024859 18435 consensus_queue.cc:260] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:52.016886 18280 leader_election.cc:341] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.024283 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282fd6d7ad0249509f67bfcbc4c0adf5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:52.032164 18435 ts_tablet_manager.cc:1398] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.049s user 0.003s sys 0.003s
I20240502 01:49:52.032871 18435 tablet_bootstrap.cc:492] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:52.034011 18313 raft_consensus.cc:2379] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:52.034682 18614 leader_election.cc:288] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.033552 18201 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.040940 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:52.043351 18233 raft_consensus.cc:3037] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.044173 18620 leader_election.cc:288] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.045217 18683 raft_consensus.cc:3037] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.046659 18238 raft_consensus.cc:2454] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:52.049193 18608 raft_consensus.cc:504] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.051189 18608 leader_election.cc:288] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.037971 18678 consensus_queue.cc:1035] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240502 01:49:52.055895 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:52.056520 18237 raft_consensus.cc:3037] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.056967 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:52.058627 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:52.059011 18233 raft_consensus.cc:2454] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:52.060226 18356 leader_election.cc:302] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:52.061268 18678 raft_consensus.cc:2781] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:49:52.061589 18280 leader_election.cc:341] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.061725 18678 raft_consensus.cc:686] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:52.062590 18678 consensus_queue.cc:237] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:52.063417 18203 leader_election.cc:341] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.064007 18356 leader_election.cc:302] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:52.063001 18237 raft_consensus.cc:2454] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
W20240502 01:49:52.072643 18355 leader_election.cc:341] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.073855 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
W20240502 01:49:52.076594 18355 leader_election.cc:341] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.065191 18493 raft_consensus.cc:2781] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:52.077128 18683 raft_consensus.cc:2726] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.089182 18433 tablet_bootstrap.cc:492] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:52.090426 18433 ts_tablet_manager.cc:1367] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.176s user 0.018s sys 0.007s
I20240502 01:49:52.093773 18435 tablet_bootstrap.cc:654] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:52.097968 18355 consensus_peers.cc:485] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:52.082648 18493 raft_consensus.cc:686] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:52.100912 18201 leader_election.cc:302] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.085430 18280 leader_election.cc:302] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.107779 18683 raft_consensus.cc:2726] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.110147 18622 raft_consensus.cc:2781] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:52.110750 18622 raft_consensus.cc:686] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:52.111845 18622 consensus_queue.cc:237] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.111119 18493 consensus_queue.cc:237] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.113236 18433 raft_consensus.cc:348] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.122489 18433 raft_consensus.cc:374] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.122828 18433 raft_consensus.cc:729] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.123096 18682 consensus_queue.cc:1035] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:52.123533 18433 consensus_queue.cc:260] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.125777 18435 tablet_bootstrap.cc:492] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:52.126156 18435 ts_tablet_manager.cc:1367] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.000s
I20240502 01:49:52.126225 18433 ts_tablet_manager.cc:1398] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20240502 01:49:52.127202 18433 tablet_bootstrap.cc:492] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:52.128176 18435 raft_consensus.cc:348] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.128695 18435 raft_consensus.cc:374] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.128906 18435 raft_consensus.cc:729] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.129413 18435 consensus_queue.cc:260] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.130607 18435 ts_tablet_manager.cc:1398] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:49:52.131539 18435 tablet_bootstrap.cc:492] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:52.143213 18433 tablet_bootstrap.cc:654] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.157920 18434 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.158469 18434 ts_tablet_manager.cc:1367] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.392s user 0.021s sys 0.046s
I20240502 01:49:52.158180 18682 consensus_queue.cc:1035] T ba7dd74b7ba34a7dab5ada581af71795 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 01:49:52.165872 18678 raft_consensus.cc:482] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.166402 18678 raft_consensus.cc:504] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.166442 18129 catalog_manager.cc:5484] T bac68a1674ab4b15a8274884f20a0266 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:52.176656 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:52.177399 18313 raft_consensus.cc:3037] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.206406 18614 raft_consensus.cc:482] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.206965 18614 raft_consensus.cc:504] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.209022 18435 tablet_bootstrap.cc:654] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.209313 18683 raft_consensus.cc:482] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.209775 18683 raft_consensus.cc:504] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.213201 18683 leader_election.cc:288] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.217909 18434 raft_consensus.cc:348] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.218742 18434 raft_consensus.cc:374] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.219017 18434 raft_consensus.cc:729] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.224198 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.219641 18434 consensus_queue.cc:260] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.208973 18614 leader_election.cc:288] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.227900 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.229719 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.230693 18387 raft_consensus.cc:2454] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:52.231887 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.232124 18203 leader_election.cc:302] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:52.232694 18238 raft_consensus.cc:2379] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:52.233088 18608 raft_consensus.cc:2781] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.233472 18608 raft_consensus.cc:482] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.234436 18608 raft_consensus.cc:3037] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:52.235528 18356 leader_election.cc:341] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.241881 18608 raft_consensus.cc:504] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.232215 18678 leader_election.cc:288] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.243975 18313 raft_consensus.cc:2454] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
W20240502 01:49:52.245741 18355 leader_election.cc:341] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.246209 18355 leader_election.cc:302] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.249061 18201 leader_election.cc:302] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.250424 18683 raft_consensus.cc:2781] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:52.250174 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.250958 18683 raft_consensus.cc:686] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:52.251549 18678 raft_consensus.cc:2726] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.250136 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d668f31e144969aae9c1410dd481e6" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.252619 18313 raft_consensus.cc:2379] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:52.253427 18434 ts_tablet_manager.cc:1398] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.095s user 0.009s sys 0.011s
I20240502 01:49:52.254480 18434 tablet_bootstrap.cc:492] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.253047 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:52.255581 18387 raft_consensus.cc:3037] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.251776 18683 consensus_queue.cc:237] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.259833 18355 leader_election.cc:302] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
W20240502 01:49:52.253077 18201 leader_election.cc:341] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.262586 18678 raft_consensus.cc:3037] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.259389 18608 leader_election.cc:288] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.270875 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:52.272742 18433 tablet_bootstrap.cc:492] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:52.273233 18433 ts_tablet_manager.cc:1367] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.146s user 0.027s sys 0.021s
I20240502 01:49:52.275586 18435 tablet_bootstrap.cc:492] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:52.283416 18435 ts_tablet_manager.cc:1367] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.152s user 0.013s sys 0.004s
W20240502 01:49:52.285415 18201 leader_election.cc:341] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.289924 18434 tablet_bootstrap.cc:654] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.276494 18433 raft_consensus.cc:348] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.294932 18433 raft_consensus.cc:374] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.295245 18433 raft_consensus.cc:729] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.279467 18387 raft_consensus.cc:2454] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:52.289703 18131 catalog_manager.cc:5484] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:52.276162 18678 raft_consensus.cc:2726] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.295933 18433 consensus_queue.cc:260] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.297060 18203 leader_election.cc:302] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.298008 18433 ts_tablet_manager.cc:1398] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20240502 01:49:52.298604 18683 raft_consensus.cc:2781] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:52.298931 18433 tablet_bootstrap.cc:492] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:52.299291 18683 raft_consensus.cc:686] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:52.300314 18683 consensus_queue.cc:237] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.305516 18433 tablet_bootstrap.cc:654] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.295624 18435 raft_consensus.cc:348] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.329398 18133 catalog_manager.cc:5484] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:52.335460 18433 tablet_bootstrap.cc:492] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:52.335595 18435 raft_consensus.cc:374] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.336046 18433 ts_tablet_manager.cc:1367] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.009s
I20240502 01:49:52.336050 18435 raft_consensus.cc:729] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.339316 18435 consensus_queue.cc:260] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.341029 18435 ts_tablet_manager.cc:1398] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.057s user 0.006s sys 0.001s
I20240502 01:49:52.341749 18435 tablet_bootstrap.cc:492] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:52.335611 18434 tablet_bootstrap.cc:492] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.344420 18434 ts_tablet_manager.cc:1367] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.000s
I20240502 01:49:52.346719 18434 raft_consensus.cc:348] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.347383 18434 raft_consensus.cc:374] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.347633 18434 raft_consensus.cc:729] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.348347 18434 consensus_queue.cc:260] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.350644 18434 ts_tablet_manager.cc:1398] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:49:52.351663 18434 tablet_bootstrap.cc:492] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.355768 18434 tablet_bootstrap.cc:654] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.343897 18433 raft_consensus.cc:348] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.356925 18433 raft_consensus.cc:374] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.357254 18433 raft_consensus.cc:729] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.358487 18433 consensus_queue.cc:260] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.366300 18433 ts_tablet_manager.cc:1398] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20240502 01:49:52.365739 18129 catalog_manager.cc:5484] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:52.368369 18129 catalog_manager.cc:5484] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:52.369701 18433 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:52.370617 18434 tablet_bootstrap.cc:492] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.371073 18434 ts_tablet_manager.cc:1367] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20240502 01:49:52.373509 18435 tablet_bootstrap.cc:654] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.373919 18434 raft_consensus.cc:348] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.374605 18434 raft_consensus.cc:374] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.374850 18434 raft_consensus.cc:729] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.375630 18434 consensus_queue.cc:260] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.376363 18433 tablet_bootstrap.cc:654] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.377943 18434 ts_tablet_manager.cc:1398] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 01:49:52.378880 18434 tablet_bootstrap.cc:492] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.390596 18434 tablet_bootstrap.cc:654] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.402195 18131 catalog_manager.cc:5484] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:52.407429 18435 tablet_bootstrap.cc:492] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:52.413619 18434 tablet_bootstrap.cc:492] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.414768 18435 ts_tablet_manager.cc:1367] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.073s user 0.012s sys 0.004s
I20240502 01:49:52.414877 18434 ts_tablet_manager.cc:1367] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.006s
I20240502 01:49:52.417656 18434 raft_consensus.cc:348] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.418402 18434 raft_consensus.cc:374] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.418058 18435 raft_consensus.cc:348] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.419077 18435 raft_consensus.cc:374] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.419292 18435 raft_consensus.cc:729] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.418706 18434 raft_consensus.cc:729] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.419831 18435 consensus_queue.cc:260] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.421448 18435 ts_tablet_manager.cc:1398] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 01:49:52.420560 18434 consensus_queue.cc:260] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.423416 18434 ts_tablet_manager.cc:1398] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240502 01:49:52.424252 18434 tablet_bootstrap.cc:492] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.429673 18434 tablet_bootstrap.cc:654] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.430481 18435 tablet_bootstrap.cc:492] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:52.437587 18623 raft_consensus.cc:482] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.438148 18623 raft_consensus.cc:504] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.440165 18623 leader_election.cc:288] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.442397 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.443354 18239 raft_consensus.cc:2379] T 0d731e8bc3f84af8aef9c511ae2e78ca P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:52.442420 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d731e8bc3f84af8aef9c511ae2e78ca" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.444551 18608 raft_consensus.cc:482] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.445081 18608 raft_consensus.cc:504] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.447093 18608 leader_election.cc:288] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.447731 18433 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:52.448233 18433 ts_tablet_manager.cc:1367] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.019s
I20240502 01:49:52.448657 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19377bfe6e48e481f6054c06e669b6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.449342 18313 raft_consensus.cc:2454] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:52.450599 18201 leader_election.cc:302] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:52.451695 18434 tablet_bootstrap.cc:492] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.451432 18433 raft_consensus.cc:348] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.452148 18434 ts_tablet_manager.cc:1367] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.008s
I20240502 01:49:52.452219 18433 raft_consensus.cc:374] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.452773 18433 raft_consensus.cc:729] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.452316 18280 leader_election.cc:302] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.453486 18433 consensus_queue.cc:260] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.453490 18608 raft_consensus.cc:2781] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.455075 18608 raft_consensus.cc:482] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.455376 18608 raft_consensus.cc:3037] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.455355 18434 raft_consensus.cc:348] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.456104 18434 raft_consensus.cc:374] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.456369 18434 raft_consensus.cc:729] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.456508 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19377bfe6e48e481f6054c06e669b6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.457129 18388 raft_consensus.cc:2454] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:52.457278 18434 consensus_queue.cc:260] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.459681 18434 ts_tablet_manager.cc:1398] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:49:52.459828 18435 tablet_bootstrap.cc:654] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.455513 18623 raft_consensus.cc:3037] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.460620 18434 tablet_bootstrap.cc:492] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.464098 18433 ts_tablet_manager.cc:1398] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20240502 01:49:52.465071 18433 tablet_bootstrap.cc:492] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:52.469578 18434 tablet_bootstrap.cc:654] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.473223 18608 raft_consensus.cc:504] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.476207 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19377bfe6e48e481f6054c06e669b6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:52.476943 18311 raft_consensus.cc:3037] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.486449 18433 tablet_bootstrap.cc:654] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.497253 18435 tablet_bootstrap.cc:492] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:52.497790 18435 ts_tablet_manager.cc:1367] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.004s
I20240502 01:49:52.499644 18623 raft_consensus.cc:2726] T 0d731e8bc3f84af8aef9c511ae2e78ca P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.500474 18435 raft_consensus.cc:348] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.501199 18435 raft_consensus.cc:374] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.501468 18435 raft_consensus.cc:729] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.502094 18435 consensus_queue.cc:260] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.504524 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19377bfe6e48e481f6054c06e669b6" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:52.497741 18311 raft_consensus.cc:2454] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:52.505252 18388 raft_consensus.cc:3037] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.504473 18435 ts_tablet_manager.cc:1398] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:49:52.506515 18605 consensus_queue.cc:1035] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:52.508558 18201 leader_election.cc:302] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:52.509672 18682 raft_consensus.cc:482] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.510249 18682 raft_consensus.cc:504] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.512360 18682 leader_election.cc:288] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.513870 18683 raft_consensus.cc:2781] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:52.514365 18683 raft_consensus.cc:686] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:52.515219 18683 consensus_queue.cc:237] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.516757 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac68a1674ab4b15a8274884f20a0266" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.518429 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac68a1674ab4b15a8274884f20a0266" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.521250 18280 leader_election.cc:302] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.522361 18682 raft_consensus.cc:2726] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.527129 18388 raft_consensus.cc:2454] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:52.529471 18608 leader_election.cc:288] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.529944 18435 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:52.530754 18434 tablet_bootstrap.cc:492] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.531222 18434 ts_tablet_manager.cc:1367] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.004s
I20240502 01:49:52.534586 18434 raft_consensus.cc:348] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.535374 18434 raft_consensus.cc:374] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.535663 18434 raft_consensus.cc:729] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.536389 18434 consensus_queue.cc:260] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.538727 18434 ts_tablet_manager.cc:1398] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240502 01:49:52.539664 18434 tablet_bootstrap.cc:492] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.541007 18129 catalog_manager.cc:5484] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:52.551779 18434 tablet_bootstrap.cc:654] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:52.557271 18315 raft_consensus.cc:3037] T bac68a1674ab4b15a8274884f20a0266 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.560302 18435 tablet_bootstrap.cc:654] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:52.564498 18280 consensus_peers.cc:485] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Couldn't send request to peer d6698cd7a2d740368c0c89233e16f04c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:52.608184 18678 raft_consensus.cc:482] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.608907 18678 raft_consensus.cc:504] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.611020 18678 leader_election.cc:288] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.617143 18433 tablet_bootstrap.cc:492] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:52.617707 18433 ts_tablet_manager.cc:1367] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.153s user 0.014s sys 0.005s
I20240502 01:49:52.620544 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.620636 18433 raft_consensus.cc:348] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.621368 18433 raft_consensus.cc:374] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 01:49:52.622100 18356 leader_election.cc:341] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.623106 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.621639 18433 raft_consensus.cc:729] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
W20240502 01:49:52.625555 18355 consensus_peers.cc:485] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:52.625649 18433 consensus_queue.cc:260] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:52.634590 18355 leader_election.cc:341] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.634935 18622 raft_consensus.cc:482] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.635052 18355 leader_election.cc:302] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.635622 18622 raft_consensus.cc:504] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.637832 18622 leader_election.cc:288] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.638411 18493 raft_consensus.cc:2726] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.639990 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.625314 18623 raft_consensus.cc:482] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.635385 18620 raft_consensus.cc:482] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.642393 18678 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.643062 18620 raft_consensus.cc:504] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.643910 18623 raft_consensus.cc:504] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.643189 18678 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.648581 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcca9e765354912af2383d0f8f8540a" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.645186 18620 leader_election.cc:288] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.649441 18388 raft_consensus.cc:2379] T 5bcca9e765354912af2383d0f8f8540a P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:52.650250 18678 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.651224 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:52.655668 18356 leader_election.cc:341] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.646297 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92fc4beef0b4747bcf45cbabe4bb118" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.652613 18280 leader_election.cc:302] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.653049 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92fc4beef0b4747bcf45cbabe4bb118" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.653592 18623 leader_election.cc:288] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:52.659926 18280 leader_election.cc:341] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.660122 18388 raft_consensus.cc:2454] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:52.661931 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.662815 18433 ts_tablet_manager.cc:1398] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.045s user 0.008s sys 0.001s
I20240502 01:49:52.663781 18433 tablet_bootstrap.cc:492] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:52.642484 18493 consensus_queue.cc:1035] T 0d731e8bc3f84af8aef9c511ae2e78ca P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20240502 01:49:52.664366 18280 leader_election.cc:302] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:52.665257 18683 raft_consensus.cc:482] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.665756 18620 raft_consensus.cc:2781] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.666158 18620 raft_consensus.cc:482] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.666478 18620 raft_consensus.cc:3037] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.665786 18683 raft_consensus.cc:504] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.673090 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b45ab88cf544fca9ca712cfaa2c601" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.661155 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e766a1489b54d248cac1c3254afcdb3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:52.687933 18493 raft_consensus.cc:482] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.688664 18493 raft_consensus.cc:504] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.688547 18237 raft_consensus.cc:2454] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:52.690815 18493 leader_election.cc:288] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.671753 18683 leader_election.cc:288] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.688268 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e766a1489b54d248cac1c3254afcdb3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.694077 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:52.695035 18355 leader_election.cc:341] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.695518 18355 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.673924 18387 raft_consensus.cc:2454] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
W20240502 01:49:52.696272 18356 leader_election.cc:341] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.696566 18493 raft_consensus.cc:2726] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.696487 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:52.695746 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b45ab88cf544fca9ca712cfaa2c601" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:52.698624 18355 leader_election.cc:341] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.699018 18355 leader_election.cc:302] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.700232 18493 raft_consensus.cc:2726] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:49:52.698169 18201 leader_election.cc:341] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.701777 18620 raft_consensus.cc:504] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.702293 18203 leader_election.cc:302] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:52.703792 18683 raft_consensus.cc:2781] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.704311 18683 raft_consensus.cc:482] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.704805 18683 raft_consensus.cc:3037] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.703943 18620 leader_election.cc:288] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.726749 18683 raft_consensus.cc:504] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.730266 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b45ab88cf544fca9ca712cfaa2c601" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
W20240502 01:49:52.731839 18201 leader_election.cc:341] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.733258 18608 raft_consensus.cc:482] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.733848 18608 raft_consensus.cc:504] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.740456 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73530ff3121b43478be31af22ba77d09" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.741293 18387 raft_consensus.cc:2454] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:52.744014 18203 leader_election.cc:302] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:52.745029 18604 raft_consensus.cc:2781] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.745424 18604 raft_consensus.cc:482] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.745752 18604 raft_consensus.cc:3037] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.748090 18608 leader_election.cc:288] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.756625 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73530ff3121b43478be31af22ba77d09" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:52.758903 18201 leader_election.cc:341] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.764148 18622 raft_consensus.cc:482] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.764782 18622 raft_consensus.cc:504] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.766963 18622 leader_election.cc:288] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.773407 18608 consensus_queue.cc:1035] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:52.660176 18623 raft_consensus.cc:3037] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.785944 18678 consensus_queue.cc:1035] T 282fd6d7ad0249509f67bfcbc4c0adf5 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:52.804919 18608 consensus_queue.cc:1035] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:52.816510 18683 leader_election.cc:288] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.816939 18604 raft_consensus.cc:504] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:52.821121 18280 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.129:38913, user_credentials={real_user=slave}} blocked reactor thread for 146568us
W20240502 01:49:52.824282 18280 leader_election.cc:341] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.824970 18280 leader_election.cc:302] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:52.827678 18622 raft_consensus.cc:2781] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:52.834915 18622 raft_consensus.cc:482] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:52.835305 18622 raft_consensus.cc:3037] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.840612 18434 tablet_bootstrap.cc:492] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:52.841194 18434 ts_tablet_manager.cc:1367] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.302s user 0.094s sys 0.104s
I20240502 01:49:52.884052 18435 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:52.841989 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92fc4beef0b4747bcf45cbabe4bb118" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:52.884698 18387 raft_consensus.cc:3037] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.885588 18435 ts_tablet_manager.cc:1367] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.356s user 0.012s sys 0.015s
I20240502 01:49:52.883248 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d66ef8312fd46c79932b145f2b6197b" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:52.900600 18434 raft_consensus.cc:348] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.901156 18434 raft_consensus.cc:374] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.901331 18434 raft_consensus.cc:729] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.901734 18434 consensus_queue.cc:260] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:52.905920 18280 leader_election.cc:341] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:52.913125 18604 leader_election.cc:288] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:52.915309 18433 tablet_bootstrap.cc:654] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:52.893810 18203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 99650.2us
I20240502 01:49:52.921669 18493 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:52.938155 18493 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.941668 18434 ts_tablet_manager.cc:1398] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.100s user 0.009s sys 0.012s
I20240502 01:49:52.925942 18623 raft_consensus.cc:2726] T 5bcca9e765354912af2383d0f8f8540a P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:52.943130 18434 tablet_bootstrap.cc:492] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:52.944975 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b45ab88cf544fca9ca712cfaa2c601" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:52.947484 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73530ff3121b43478be31af22ba77d09" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:52.963948 18435 raft_consensus.cc:348] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.964569 18435 raft_consensus.cc:374] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:52.964792 18435 raft_consensus.cc:729] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:52.965267 18435 consensus_queue.cc:260] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:52.955552 18392 raft_consensus.cc:3037] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.966234 18315 raft_consensus.cc:3037] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:52.973521 18435 ts_tablet_manager.cc:1398] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.087s user 0.005s sys 0.001s
I20240502 01:49:53.017969 18315 raft_consensus.cc:2454] T 73530ff3121b43478be31af22ba77d09 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.014218 18493 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:52.972344 18723 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.025048 18435 tablet_bootstrap.cc:492] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.028154 18728 consensus_queue.cc:1035] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:53.026535 18622 raft_consensus.cc:504] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.032506 18622 leader_election.cc:288] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.034830 18201 leader_election.cc:302] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:53.009205 18387 raft_consensus.cc:2454] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:53.035945 18728 raft_consensus.cc:2781] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:52.974325 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73530ff3121b43478be31af22ba77d09" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.036712 18728 raft_consensus.cc:686] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:53.030694 18392 raft_consensus.cc:2454] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.037693 18728 consensus_queue.cc:237] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.053177 18280 leader_election.cc:302] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:52.999596 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92fc4beef0b4747bcf45cbabe4bb118" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:53.036916 18386 raft_consensus.cc:3037] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.061182 18735 raft_consensus.cc:2781] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.061759 18735 raft_consensus.cc:686] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:53.059587 18233 raft_consensus.cc:3037] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.063093 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e766a1489b54d248cac1c3254afcdb3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.050381 18723 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.062779 18735 consensus_queue.cc:237] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.080873 18723 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:53.087422 18280 leader_election.cc:341] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.095324 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.096822 18433 tablet_bootstrap.cc:492] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.097330 18433 ts_tablet_manager.cc:1367] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.434s user 0.031s sys 0.023s
I20240502 01:49:53.095376 18434 tablet_bootstrap.cc:654] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.103466 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.104426 18315 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:53.106119 18433 raft_consensus.cc:348] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.106328 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d66ef8312fd46c79932b145f2b6197b" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.106876 18433 raft_consensus.cc:374] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.107192 18433 raft_consensus.cc:729] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.107831 18433 consensus_queue.cc:260] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.109357 18433 ts_tablet_manager.cc:1398] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20240502 01:49:53.110193 18433 tablet_bootstrap.cc:492] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.114693 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.117771 18392 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:53.124231 18733 raft_consensus.cc:482] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.124792 18733 raft_consensus.cc:504] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.128274 18733 leader_election.cc:288] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:53.130666 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e766a1489b54d248cac1c3254afcdb3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:53.131356 18280 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.125984 18237 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:53.133234 18723 raft_consensus.cc:2781] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:49:53.133634 18723 raft_consensus.cc:482] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.133947 18392 raft_consensus.cc:3037] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.133976 18723 raft_consensus.cc:3037] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:53.132164 18231 raft_consensus.cc:3037] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.153304 18723 raft_consensus.cc:504] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.153268 18231 raft_consensus.cc:2454] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:53.156493 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.157174 18235 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:53.157236 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:53.157924 18234 raft_consensus.cc:3037] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:53.158735 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e973320cf04094a84d5530177b70c3" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.159404 18388 raft_consensus.cc:3037] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:49:53.164212 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:53.167398 18234 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 2.
I20240502 01:49:53.168175 18388 raft_consensus.cc:2454] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 2.
I20240502 01:49:53.168396 18129 catalog_manager.cc:5484] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:53.169549 18738 consensus_queue.cc:1035] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:53.171116 18280 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.163045 18723 leader_election.cc:288] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 2 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.173028 18355 leader_election.cc:302] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.173751 18678 raft_consensus.cc:2741] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240502 01:49:53.174538 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.175688 18237 raft_consensus.cc:2379] T 45daae4b488442c797a69bb638169fc4 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:53.186745 18386 raft_consensus.cc:2454] T 73530ff3121b43478be31af22ba77d09 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.187222 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45daae4b488442c797a69bb638169fc4" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.172504 18723 raft_consensus.cc:2781] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:53.194422 18723 raft_consensus.cc:686] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 2 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:53.216292 18355 leader_election.cc:302] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:53.218122 18435 tablet_bootstrap.cc:654] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.219461 18605 raft_consensus.cc:2726] T 45daae4b488442c797a69bb638169fc4 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:53.218963 18723 consensus_queue.cc:237] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.204519 18233 raft_consensus.cc:2454] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
W20240502 01:49:53.228332 18280 leader_election.cc:341] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.228901 18280 leader_election.cc:302] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:53.232837 18736 raft_consensus.cc:2726] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:53.247602 18433 tablet_bootstrap.cc:654] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.252835 18203 leader_election.cc:302] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:53.254160 18738 raft_consensus.cc:2781] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.254659 18738 raft_consensus.cc:686] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:53.255156 18608 consensus_queue.cc:1035] T 5bcca9e765354912af2383d0f8f8540a P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:53.255489 18738 consensus_queue.cc:237] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.268769 18280 leader_election.cc:302] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:53.269884 18735 raft_consensus.cc:2781] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.270488 18735 raft_consensus.cc:686] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:53.271579 18735 consensus_queue.cc:237] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.277899 18608 consensus_queue.cc:1035] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:53.282784 18682 consensus_queue.cc:1035] T 45daae4b488442c797a69bb638169fc4 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:53.295006 18434 tablet_bootstrap.cc:492] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:53.295486 18434 ts_tablet_manager.cc:1367] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.353s user 0.013s sys 0.021s
I20240502 01:49:53.297948 18434 raft_consensus.cc:348] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.298636 18434 raft_consensus.cc:374] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.298902 18434 raft_consensus.cc:729] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.300197 18434 consensus_queue.cc:260] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.328612 18434 ts_tablet_manager.cc:1398] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20240502 01:49:53.338173 18433 tablet_bootstrap.cc:492] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.338696 18433 ts_tablet_manager.cc:1367] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.229s user 0.010s sys 0.005s
I20240502 01:49:53.339712 18435 tablet_bootstrap.cc:492] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:53.340128 18435 ts_tablet_manager.cc:1367] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.320s user 0.014s sys 0.006s
I20240502 01:49:53.347743 18433 raft_consensus.cc:348] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.348977 18433 raft_consensus.cc:374] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.350409 18433 raft_consensus.cc:729] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.351068 18433 consensus_queue.cc:260] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.352789 18435 raft_consensus.cc:348] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.353443 18433 ts_tablet_manager.cc:1398] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20240502 01:49:53.354431 18433 tablet_bootstrap.cc:492] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.353526 18435 raft_consensus.cc:374] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.356810 18435 raft_consensus.cc:729] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.357447 18435 consensus_queue.cc:260] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.359964 18435 ts_tablet_manager.cc:1398] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 01:49:53.360316 18434 tablet_bootstrap.cc:492] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:53.360896 18435 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.375985 18435 tablet_bootstrap.cc:654] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.399689 18434 tablet_bootstrap.cc:654] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.404515 18131 catalog_manager.cc:5484] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:53.405443 18433 tablet_bootstrap.cc:654] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.421226 18434 tablet_bootstrap.cc:492] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:53.421641 18434 ts_tablet_manager.cc:1367] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.004s
I20240502 01:49:53.423655 18434 raft_consensus.cc:348] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.423786 18133 catalog_manager.cc:5484] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 2, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 2 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:53.425173 18133 catalog_manager.cc:5484] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:53.432273 18435 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:53.432756 18435 ts_tablet_manager.cc:1367] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.010s
I20240502 01:49:53.434859 18435 raft_consensus.cc:348] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.435560 18435 raft_consensus.cc:374] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.435850 18435 raft_consensus.cc:729] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.436491 18435 consensus_queue.cc:260] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.437837 18433 tablet_bootstrap.cc:492] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.438313 18433 ts_tablet_manager.cc:1367] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.006s
I20240502 01:49:53.438977 18435 ts_tablet_manager.cc:1398] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 01:49:53.439968 18435 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.440963 18433 raft_consensus.cc:348] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.441656 18433 raft_consensus.cc:374] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.441938 18433 raft_consensus.cc:729] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.442576 18433 consensus_queue.cc:260] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.424226 18434 raft_consensus.cc:374] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.443517 18434 raft_consensus.cc:729] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.444079 18314 raft_consensus.cc:3037] T 282fd6d7ad0249509f67bfcbc4c0adf5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.445735 18433 ts_tablet_manager.cc:1398] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 01:49:53.446414 18433 tablet_bootstrap.cc:492] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.446914 18131 catalog_manager.cc:5484] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:53.444273 18434 consensus_queue.cc:260] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.453975 18434 ts_tablet_manager.cc:1398] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.032s user 0.004s sys 0.001s
I20240502 01:49:53.454859 18434 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:53.456916 18435 tablet_bootstrap.cc:654] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.451849 18433 tablet_bootstrap.cc:654] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.460238 18434 tablet_bootstrap.cc:654] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.494086 18435 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:53.494611 18435 ts_tablet_manager.cc:1367] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.010s
I20240502 01:49:53.496898 18434 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:53.497435 18434 ts_tablet_manager.cc:1367] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.006s
I20240502 01:49:53.497668 18435 raft_consensus.cc:348] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.498399 18435 raft_consensus.cc:374] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.498677 18435 raft_consensus.cc:729] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.499308 18435 consensus_queue.cc:260] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.501013 18435 ts_tablet_manager.cc:1398] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240502 01:49:53.501730 18435 tablet_bootstrap.cc:492] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.504401 18434 raft_consensus.cc:348] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.505172 18434 raft_consensus.cc:374] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.506884 18435 tablet_bootstrap.cc:654] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.509559 18433 tablet_bootstrap.cc:492] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.510107 18434 raft_consensus.cc:729] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.510445 18433 ts_tablet_manager.cc:1367] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.064s user 0.017s sys 0.006s
I20240502 01:49:53.510977 18434 consensus_queue.cc:260] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.512989 18433 raft_consensus.cc:348] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.514254 18433 raft_consensus.cc:374] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.514467 18433 raft_consensus.cc:729] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.515067 18433 consensus_queue.cc:260] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.516409 18434 ts_tablet_manager.cc:1398] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240502 01:49:53.517333 18434 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:53.521792 18433 ts_tablet_manager.cc:1398] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:49:53.522727 18433 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.525823 18434 tablet_bootstrap.cc:654] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.530025 18435 tablet_bootstrap.cc:492] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:53.530642 18435 ts_tablet_manager.cc:1367] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20240502 01:49:53.530704 18433 tablet_bootstrap.cc:654] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.534840 18435 raft_consensus.cc:348] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.535565 18435 raft_consensus.cc:374] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.535842 18435 raft_consensus.cc:729] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.536458 18435 consensus_queue.cc:260] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.538465 18435 ts_tablet_manager.cc:1398] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240502 01:49:53.539681 18435 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.542588 18433 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.543056 18433 ts_tablet_manager.cc:1367] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20240502 01:49:53.545626 18608 raft_consensus.cc:482] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.545420 18433 raft_consensus.cc:348] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.546073 18433 raft_consensus.cc:374] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.546520 18433 raft_consensus.cc:729] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.546057 18608 raft_consensus.cc:504] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.547107 18433 consensus_queue.cc:260] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.548831 18435 tablet_bootstrap.cc:654] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.549351 18608 leader_election.cc:288] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.549556 18433 ts_tablet_manager.cc:1398] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 01:49:53.550452 18433 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.550740 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.551328 18388 raft_consensus.cc:2454] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:53.552675 18203 leader_election.cc:302] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.553426 18728 raft_consensus.cc:2781] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.553758 18728 raft_consensus.cc:482] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.554062 18728 raft_consensus.cc:3037] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.555653 18433 tablet_bootstrap.cc:654] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.558774 18727 raft_consensus.cc:482] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.560149 18727 raft_consensus.cc:504] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.562218 18727 leader_election.cc:288] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.564270 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be322d2860f40408a8cd8199731477c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.565306 18392 raft_consensus.cc:2454] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:53.566808 18203 leader_election.cc:302] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.569276 18435 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:53.569783 18433 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.570681 18738 consensus_queue.cc:1035] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:53.571040 18435 ts_tablet_manager.cc:1367] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.005s
I20240502 01:49:53.571079 18433 ts_tablet_manager.cc:1367] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.007s
I20240502 01:49:53.574079 18433 raft_consensus.cc:348] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.574785 18433 raft_consensus.cc:374] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.575060 18433 raft_consensus.cc:729] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.569590 18727 raft_consensus.cc:2781] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.576716 18727 raft_consensus.cc:482] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.577039 18727 raft_consensus.cc:3037] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.578827 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.582842 18433 consensus_queue.cc:260] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.585143 18614 raft_consensus.cc:482] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.585491 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be322d2860f40408a8cd8199731477c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.585678 18614 raft_consensus.cc:504] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:53.586252 18201 leader_election.cc:341] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.586755 18435 raft_consensus.cc:348] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.587460 18435 raft_consensus.cc:374] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.587803 18614 leader_election.cc:288] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:53.587471 18727 raft_consensus.cc:504] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.589349 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8936a88f18dd4d1185402793335458f2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.590047 18233 raft_consensus.cc:2454] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:53.587836 18435 raft_consensus.cc:729] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.590739 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be322d2860f40408a8cd8199731477c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.591297 18356 leader_election.cc:302] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.586755 18433 ts_tablet_manager.cc:1398] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.015s user 0.000s sys 0.004s
I20240502 01:49:53.591449 18435 consensus_queue.cc:260] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.591382 18392 raft_consensus.cc:3037] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.592842 18433 tablet_bootstrap.cc:492] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.592895 18608 raft_consensus.cc:482] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.593717 18435 ts_tablet_manager.cc:1398] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240502 01:49:53.593396 18608 raft_consensus.cc:504] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.594871 18435 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.595127 18736 raft_consensus.cc:482] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.595584 18736 raft_consensus.cc:504] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.596202 18614 raft_consensus.cc:2781] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.596582 18614 raft_consensus.cc:482] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.596908 18614 raft_consensus.cc:3037] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.597026 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d66ef8312fd46c79932b145f2b6197b" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.597929 18388 raft_consensus.cc:2454] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:53.598399 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.599025 18203 leader_election.cc:302] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
W20240502 01:49:53.589483 18201 leader_election.cc:341] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:53.600127 18280 leader_election.cc:341] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.599711 18738 raft_consensus.cc:2781] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.600798 18608 leader_election.cc:288] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.601061 18738 raft_consensus.cc:482] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.601514 18738 raft_consensus.cc:3037] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.603160 18736 leader_election.cc:288] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.604635 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.605444 18388 raft_consensus.cc:2454] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:53.593832 18727 leader_election.cc:288] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.606674 18280 leader_election.cc:302] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:53.607470 18435 tablet_bootstrap.cc:654] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.608705 18736 raft_consensus.cc:2781] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.613402 18736 raft_consensus.cc:482] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.614146 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8936a88f18dd4d1185402793335458f2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.613895 18736 raft_consensus.cc:3037] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.615631 18733 raft_consensus.cc:482] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 01:49:53.617295 18355 leader_election.cc:341] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.623520 18433 tablet_bootstrap.cc:654] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.623216 18392 raft_consensus.cc:2454] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.616637 18733 raft_consensus.cc:504] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.609315 18728 raft_consensus.cc:504] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.627847 18203 leader_election.cc:302] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.629930 18727 raft_consensus.cc:482] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.635155 18608 raft_consensus.cc:2781] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.627888 18614 raft_consensus.cc:504] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.641788 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be322d2860f40408a8cd8199731477c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:53.646510 18608 raft_consensus.cc:686] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:53.642216 18727 raft_consensus.cc:504] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.668475 18733 leader_election.cc:288] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:53.657169 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d66ef8312fd46c79932b145f2b6197b" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.669914 18727 leader_election.cc:288] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.670037 18314 raft_consensus.cc:2454] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:53.672127 18738 raft_consensus.cc:504] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.673460 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e766a1489b54d248cac1c3254afcdb3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.674126 18738 leader_election.cc:288] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.677155 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.677269 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d66ef8312fd46c79932b145f2b6197b" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.677903 18386 raft_consensus.cc:3037] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.674263 18233 raft_consensus.cc:2379] T 0e766a1489b54d248cac1c3254afcdb3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:53.677870 18388 raft_consensus.cc:2454] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:53.680620 18435 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:53.681879 18435 ts_tablet_manager.cc:1367] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.087s user 0.020s sys 0.006s
I20240502 01:49:53.681830 18203 leader_election.cc:302] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
W20240502 01:49:53.683171 18201 leader_election.cc:341] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.665616 18620 raft_consensus.cc:482] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.684095 18620 raft_consensus.cc:504] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.686188 18620 leader_election.cc:288] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.686856 18608 consensus_queue.cc:237] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.683127 18434 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:53.692459 18434 ts_tablet_manager.cc:1367] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.175s user 0.029s sys 0.036s
I20240502 01:49:53.695303 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:53.718225 18727 raft_consensus.cc:2781] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.776417 18727 raft_consensus.cc:482] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.776594 18392 raft_consensus.cc:2454] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:53.765538 18614 leader_election.cc:288] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:53.674407 18725 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:53.782208 18280 leader_election.cc:302] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.783315 18233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.783823 18725 raft_consensus.cc:2781] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:53.784106 18728 leader_election.cc:288] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.766727 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d66ef8312fd46c79932b145f2b6197b" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:53.784880 18312 raft_consensus.cc:3037] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.786187 18232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8936a88f18dd4d1185402793335458f2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:53.787061 18232 raft_consensus.cc:3037] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.772390 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:53.774574 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e766a1489b54d248cac1c3254afcdb3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.776861 18727 raft_consensus.cc:3037] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.721705 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.768666 18435 raft_consensus.cc:348] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.784390 18725 raft_consensus.cc:482] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:53.792098 18725 raft_consensus.cc:3037] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.783982 18623 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:53.792025 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.796473 18389 raft_consensus.cc:3037] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.797165 18435 raft_consensus.cc:374] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.797502 18435 raft_consensus.cc:729] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.798143 18435 consensus_queue.cc:260] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.800901 18312 raft_consensus.cc:2454] T 8d66ef8312fd46c79932b145f2b6197b P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.766130 18433 tablet_bootstrap.cc:492] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:53.802008 18433 ts_tablet_manager.cc:1367] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.209s user 0.017s sys 0.019s
I20240502 01:49:53.799742 18435 ts_tablet_manager.cc:1398] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.117s user 0.011s sys 0.007s
I20240502 01:49:53.795759 18727 raft_consensus.cc:504] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.805616 18233 raft_consensus.cc:2379] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:53.810228 18725 raft_consensus.cc:504] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.807204 18131 catalog_manager.cc:5484] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:53.811666 18727 leader_election.cc:288] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.817780 18386 raft_consensus.cc:2454] T 8d66ef8312fd46c79932b145f2b6197b P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.828830 18435 tablet_bootstrap.cc:492] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:53.833750 18434 raft_consensus.cc:348] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.834515 18434 raft_consensus.cc:374] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.834810 18434 raft_consensus.cc:729] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.836175 18314 raft_consensus.cc:2379] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:53.837926 18203 leader_election.cc:302] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.840987 18434 consensus_queue.cc:260] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.763408 18736 raft_consensus.cc:504] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.843853 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8936a88f18dd4d1185402793335458f2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:53.840896 18727 consensus_queue.cc:1035] T 73530ff3121b43478be31af22ba77d09 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 01:49:53.842732 18738 consensus_queue.cc:1035] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240502 01:49:53.842540 18683 raft_consensus.cc:2781] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.852697 18683 raft_consensus.cc:686] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:53.853790 18683 consensus_queue.cc:237] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.853848 18433 raft_consensus.cc:348] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.855599 18433 raft_consensus.cc:374] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.855829 18433 raft_consensus.cc:729] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.856456 18433 consensus_queue.cc:260] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.845768 18434 ts_tablet_manager.cc:1398] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.153s user 0.010s sys 0.003s
I20240502 01:49:53.845921 18604 raft_consensus.cc:482] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.857909 18434 tablet_bootstrap.cc:492] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:53.857626 18604 raft_consensus.cc:504] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.854872 18389 raft_consensus.cc:2454] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:53.862448 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:53.863255 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
W20240502 01:49:53.862538 18201 consensus_peers.cc:485] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:53.863315 18234 raft_consensus.cc:2379] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:53.864771 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c0d6ed72f045a3b39631b83a054d78" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
W20240502 01:49:53.865581 18201 leader_election.cc:341] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.860342 18604 leader_election.cc:288] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.862524 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.866513 18391 raft_consensus.cc:3037] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.866624 18605 raft_consensus.cc:482] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:53.867347 18605 raft_consensus.cc:504] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.869859 18605 leader_election.cc:288] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:53.870374 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084a332369434663a3a306d0fb57e927" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:53.870958 18355 leader_election.cc:302] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
W20240502 01:49:53.871495 18356 leader_election.cc:341] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.871945 18605 raft_consensus.cc:3037] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:53.873416 18280 leader_election.cc:341] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.875425 18391 raft_consensus.cc:2454] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:53.846343 18232 raft_consensus.cc:2454] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:53.876866 18203 leader_election.cc:302] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:53.860293 18725 leader_election.cc:288] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.880252 18433 ts_tablet_manager.cc:1398] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
W20240502 01:49:53.880968 18203 leader_election.cc:341] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:53.879586 18233 raft_consensus.cc:1261] T 97e973320cf04094a84d5530177b70c3 P 26c26643de084d6b8acaff73fa67f794 [term 2 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:49:53.879792 18608 raft_consensus.cc:2781] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.884485 18608 raft_consensus.cc:686] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:53.885392 18608 consensus_queue.cc:237] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.888075 18280 leader_election.cc:302] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:53.891455 18725 raft_consensus.cc:2781] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.891959 18725 raft_consensus.cc:686] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:53.892877 18725 consensus_queue.cc:237] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:53.897397 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.898370 18386 raft_consensus.cc:2379] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate 98f3bf374d49481cb6ebe0672e5b81dc in this term.
I20240502 01:49:53.898918 18736 leader_election.cc:288] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:53.905216 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5411115a92043f684df0c01d3cc5e36" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:53.906349 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c0d6ed72f045a3b39631b83a054d78" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:53.907451 18356 leader_election.cc:302] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:53.909332 18433 tablet_bootstrap.cc:492] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:53.907073 18315 raft_consensus.cc:2454] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:53.927325 18724 consensus_queue.cc:1035] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20240502 01:49:53.985121 18434 tablet_bootstrap.cc:654] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.922116 18736 consensus_queue.cc:1035] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20240502 01:49:53.990375 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:53.991149 18385 raft_consensus.cc:3037] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.992841 18435 tablet_bootstrap.cc:654] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.987377 18386 raft_consensus.cc:1261] T 97e973320cf04094a84d5530177b70c3 P d6698cd7a2d740368c0c89233e16f04c [term 2 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:49:53.978322 18605 raft_consensus.cc:2726] T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.028925 18734 raft_consensus.cc:2781] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:54.029521 18734 raft_consensus.cc:686] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:54.030362 18734 consensus_queue.cc:237] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.063361 18433 tablet_bootstrap.cc:654] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:54.087458 18355 leader_election.cc:341] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.075969 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084a332369434663a3a306d0fb57e927" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.118142 18435 tablet_bootstrap.cc:492] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:54.118934 18435 ts_tablet_manager.cc:1367] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.290s user 0.015s sys 0.006s
I20240502 01:49:54.088698 18201 leader_election.cc:302] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
W20240502 01:49:54.122841 18201 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 45959.8us
I20240502 01:49:54.125010 18434 tablet_bootstrap.cc:492] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:54.125571 18434 ts_tablet_manager.cc:1367] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.268s user 0.024s sys 0.001s
I20240502 01:49:54.126942 18131 catalog_manager.cc:5484] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
W20240502 01:49:54.129642 18280 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.129:38913, user_credentials={real_user=slave}} blocked reactor thread for 58512us
W20240502 01:49:54.131789 18356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.129:38913, user_credentials={real_user=slave}} blocked reactor thread for 44992.8us
W20240502 01:49:54.137161 18355 leader_election.cc:341] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.137676 18355 leader_election.cc:302] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.138919 18789 consensus_queue.cc:1035] T 97e973320cf04094a84d5530177b70c3 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 01:49:54.139330 18795 raft_consensus.cc:2726] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.144270 18435 raft_consensus.cc:348] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.144850 18435 raft_consensus.cc:374] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.145115 18435 raft_consensus.cc:729] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.145625 18435 consensus_queue.cc:260] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.128289 18604 raft_consensus.cc:2726] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.123677 18201 leader_election.cc:302] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:54.158816 18683 raft_consensus.cc:2781] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:54.159267 18683 raft_consensus.cc:482] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:54.164757 18435 ts_tablet_manager.cc:1398] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.046s user 0.007s sys 0.001s
I20240502 01:49:54.174706 18434 raft_consensus.cc:348] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.175536 18434 raft_consensus.cc:374] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.159605 18683 raft_consensus.cc:3037] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.175801 18434 raft_consensus.cc:729] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.184726 18434 consensus_queue.cc:260] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.194655 18434 ts_tablet_manager.cc:1398] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.069s user 0.006s sys 0.001s
I20240502 01:49:54.186372 18385 raft_consensus.cc:2454] T 420ebc81e61440d997f99ad838b88f79 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:54.282914 18683 raft_consensus.cc:504] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.294788 18683 leader_election.cc:288] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:54.310235 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c0d6ed72f045a3b39631b83a054d78" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
W20240502 01:49:54.315320 18203 leader_election.cc:341] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.318974 18723 consensus_queue.cc:1035] T 0e766a1489b54d248cac1c3254afcdb3 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:54.320531 18433 tablet_bootstrap.cc:492] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:54.321035 18433 ts_tablet_manager.cc:1367] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.413s user 0.019s sys 0.016s
I20240502 01:49:54.320556 18280 leader_election.cc:302] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:54.323204 18773 raft_consensus.cc:482] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.323750 18725 raft_consensus.cc:2781] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:54.323722 18773 raft_consensus.cc:504] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.324702 18725 raft_consensus.cc:686] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:54.329563 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c0d6ed72f045a3b39631b83a054d78" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:54.332191 18313 raft_consensus.cc:3037] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.325596 18725 consensus_queue.cc:237] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.339835 18433 raft_consensus.cc:348] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.340497 18433 raft_consensus.cc:374] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.340767 18433 raft_consensus.cc:729] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.347095 18773 leader_election.cc:288] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:54.352481 18738 consensus_queue.cc:1035] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:54.351310 18433 consensus_queue.cc:260] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.356459 18724 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:54.357723 18433 ts_tablet_manager.cc:1398] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20240502 01:49:54.358701 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:54.361342 18433 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
W20240502 01:49:54.364189 18356 leader_election.cc:341] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.335619 18776 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 01:49:54.374245 18435 tablet_bootstrap.cc:492] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:54.378134 18604 consensus_queue.cc:1035] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:54.385349 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.377403 18727 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 01:49:54.387099 18434 tablet_bootstrap.cc:492] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
W20240502 01:49:54.388391 18201 consensus_peers.cc:485] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:54.376016 18773 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.420624 18800 raft_consensus.cc:482] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.489244 18604 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.389925 18129 catalog_manager.cc:5484] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:54.489794 18604 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.488715 18800 raft_consensus.cc:504] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.491485 18129 catalog_manager.cc:5484] T 8d66ef8312fd46c79932b145f2b6197b P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:54.474462 18773 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.518704 18604 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:54.527184 18355 leader_election.cc:341] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:54.509111 18201 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 99219us
I20240502 01:49:54.533428 18800 leader_election.cc:288] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:54.527679 18355 leader_election.cc:302] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.539392 18724 raft_consensus.cc:482] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.539961 18724 raft_consensus.cc:504] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.546181 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd8414d528f4407b7369da924f707c5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:54.547605 18433 tablet_bootstrap.cc:654] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.551784 18724 leader_election.cc:288] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:54.556372 18309 raft_consensus.cc:3037] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:54.558063 18356 leader_election.cc:341] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.562737 18773 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:54.562826 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8936a88f18dd4d1185402793335458f2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:54.565680 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd8414d528f4407b7369da924f707c5" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.547987 18734 raft_consensus.cc:2726] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.569010 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.571664 18131 catalog_manager.cc:5484] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:54.573246 18313 raft_consensus.cc:2454] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:54.579877 18435 tablet_bootstrap.cc:654] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.648872 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8936a88f18dd4d1185402793335458f2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:54.655483 18201 leader_election.cc:302] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters: d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:54.657384 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.657162 18727 raft_consensus.cc:2781] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:49:54.662799 18201 leader_election.cc:341] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:54.666692 18355 leader_election.cc:341] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:54.667369 18355 leader_election.cc:341] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.667745 18355 leader_election.cc:302] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.671124 18734 raft_consensus.cc:2726] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.672515 18280 leader_election.cc:302] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
W20240502 01:49:54.677100 18201 consensus_peers.cc:485] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 -> Peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Couldn't send request to peer 98f3bf374d49481cb6ebe0672e5b81dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:54.687798 18434 tablet_bootstrap.cc:654] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.674448 18727 raft_consensus.cc:686] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:54.696830 18727 consensus_queue.cc:237] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:54.698132 18814 log.cc:924] Time spent T 0e766a1489b54d248cac1c3254afcdb3 P d6698cd7a2d740368c0c89233e16f04c: Append to log took a long time: real 0.083s user 0.007s sys 0.009s
W20240502 01:49:54.701879 18808 log.cc:924] Time spent T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c: Append to log took a long time: real 0.057s user 0.000s sys 0.014s
I20240502 01:49:54.673015 18802 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20240502 01:49:54.715960 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:54.716509 18620 raft_consensus.cc:2726] T 8936a88f18dd4d1185402793335458f2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.670917 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:54.717515 18237 raft_consensus.cc:2454] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:54.716599 18392 raft_consensus.cc:2454] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:54.721506 18356 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
W20240502 01:49:54.725783 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:54.729264 18798 raft_consensus.cc:2781] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:54.730840 18798 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:54.729900 18614 consensus_queue.cc:1035] T 8936a88f18dd4d1185402793335458f2 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:54.734722 18798 raft_consensus.cc:3037] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.737844 18433 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:54.738348 18433 ts_tablet_manager.cc:1367] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.377s user 0.033s sys 0.038s
I20240502 01:49:54.750989 18433 raft_consensus.cc:348] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.752290 18433 raft_consensus.cc:374] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.752638 18433 raft_consensus.cc:729] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.753509 18433 consensus_queue.cc:260] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.762610 18433 ts_tablet_manager.cc:1398] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.024s user 0.007s sys 0.001s
I20240502 01:49:54.768461 18433 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:54.781884 18131 catalog_manager.cc:5484] T 8936a88f18dd4d1185402793335458f2 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } }
W20240502 01:49:54.809834 18203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 51485.8us
I20240502 01:49:54.811584 18203 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.813335 18435 tablet_bootstrap.cc:492] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:54.813967 18435 ts_tablet_manager.cc:1367] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.440s user 0.027s sys 0.020s
I20240502 01:49:54.817137 18738 raft_consensus.cc:2781] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:54.819298 18738 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:54.830348 18433 tablet_bootstrap.cc:654] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.830710 18738 raft_consensus.cc:3037] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.832042 18798 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.842685 18798 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:54.848362 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:54.852823 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
W20240502 01:49:54.858415 18355 leader_election.cc:341] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.874491 18723 consensus_queue.cc:1035] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:54.879993 18434 tablet_bootstrap.cc:492] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:54.880465 18434 ts_tablet_manager.cc:1367] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.494s user 0.015s sys 0.026s
I20240502 01:49:54.883962 18434 raft_consensus.cc:348] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.884650 18434 raft_consensus.cc:374] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.884936 18434 raft_consensus.cc:729] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.884889 18129 catalog_manager.cc:5484] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:54.886128 18434 consensus_queue.cc:260] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.889659 18434 ts_tablet_manager.cc:1398] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:49:54.890610 18434 tablet_bootstrap.cc:492] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
W20240502 01:49:54.893388 18280 consensus_peers.cc:485] T 420ebc81e61440d997f99ad838b88f79 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Couldn't send request to peer 26c26643de084d6b8acaff73fa67f794. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:54.897521 18435 raft_consensus.cc:348] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.887435 18738 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.905388 18234 raft_consensus.cc:2379] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:54.909380 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:54.898286 18435 raft_consensus.cc:374] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 01:49:54.911118 18201 leader_election.cc:341] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.911406 18356 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.914542 18434 tablet_bootstrap.cc:654] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.912519 18433 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:54.910835 18435 raft_consensus.cc:729] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.915242 18433 ts_tablet_manager.cc:1367] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.151s user 0.014s sys 0.001s
I20240502 01:49:54.915930 18738 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:54.916894 18435 consensus_queue.cc:260] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.917834 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:54.918355 18433 raft_consensus.cc:348] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.918947 18433 raft_consensus.cc:374] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.919152 18433 raft_consensus.cc:729] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.914450 18773 raft_consensus.cc:2726] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.922781 18390 raft_consensus.cc:2379] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26c26643de084d6b8acaff73fa67f794 in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:54.925390 18203 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:54.926093 18608 raft_consensus.cc:2726] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.932039 18773 raft_consensus.cc:482] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.932765 18773 raft_consensus.cc:504] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.936631 18773 leader_election.cc:288] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:54.936744 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c0d6ed72f045a3b39631b83a054d78" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:54.939666 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c0d6ed72f045a3b39631b83a054d78" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.940696 18435 ts_tablet_manager.cc:1398] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.126s user 0.006s sys 0.001s
I20240502 01:49:54.944164 18310 raft_consensus.cc:2379] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:54.946205 18355 leader_election.cc:302] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.947028 18773 raft_consensus.cc:3037] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.944135 18433 consensus_queue.cc:260] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.950685 18433 ts_tablet_manager.cc:1398] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.035s user 0.002s sys 0.004s
I20240502 01:49:54.951571 18433 tablet_bootstrap.cc:492] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:54.953091 18435 tablet_bootstrap.cc:492] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:54.956782 18773 raft_consensus.cc:2726] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.962874 18433 tablet_bootstrap.cc:654] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.964323 18773 raft_consensus.cc:482] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.965039 18435 tablet_bootstrap.cc:654] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.965006 18773 raft_consensus.cc:504] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.967969 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a4278616b44d738b2b7aa7eafef008" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:54.969646 18356 leader_election.cc:341] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.970629 18773 leader_election.cc:288] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:54.973026 18435 tablet_bootstrap.cc:492] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:54.973489 18435 ts_tablet_manager.cc:1367] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20240502 01:49:54.973780 18433 tablet_bootstrap.cc:492] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:54.974220 18433 ts_tablet_manager.cc:1367] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.003s
I20240502 01:49:54.974124 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a4278616b44d738b2b7aa7eafef008" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.975955 18435 raft_consensus.cc:348] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.976661 18435 raft_consensus.cc:374] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 01:49:54.976682 18355 leader_election.cc:341] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:54.976595 18433 raft_consensus.cc:348] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.977154 18435 raft_consensus.cc:729] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.977448 18433 raft_consensus.cc:374] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.977502 18355 leader_election.cc:302] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:54.977824 18433 raft_consensus.cc:729] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.978503 18773 raft_consensus.cc:2726] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:54.977874 18435 consensus_queue.cc:260] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.978505 18433 consensus_queue.cc:260] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.980309 18435 ts_tablet_manager.cc:1398] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 01:49:54.981022 18433 ts_tablet_manager.cc:1398] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:49:54.981829 18433 tablet_bootstrap.cc:492] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:54.982749 18435 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:54.985569 18608 raft_consensus.cc:482] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.987354 18608 raft_consensus.cc:504] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.988224 18433 tablet_bootstrap.cc:654] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.988795 18434 tablet_bootstrap.cc:492] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:54.989217 18434 ts_tablet_manager.cc:1367] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.099s user 0.021s sys 0.013s
I20240502 01:49:54.989408 18608 leader_election.cc:288] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:54.990350 18435 tablet_bootstrap.cc:654] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.991204 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd8414d528f4407b7369da924f707c5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:54.992095 18388 raft_consensus.cc:2454] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:54.992871 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd8414d528f4407b7369da924f707c5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:54.993513 18310 raft_consensus.cc:2454] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:54.993810 18203 leader_election.cc:302] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:54.994738 18608 raft_consensus.cc:2781] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:54.995241 18608 raft_consensus.cc:482] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:54.991664 18434 raft_consensus.cc:348] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:54.995728 18608 raft_consensus.cc:3037] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.996239 18434 raft_consensus.cc:374] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.996608 18434 raft_consensus.cc:729] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.997362 18434 consensus_queue.cc:260] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.002097 18434 ts_tablet_manager.cc:1398] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:49:55.003170 18434 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.006727 18435 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.006759 18433 tablet_bootstrap.cc:492] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:55.007375 18435 ts_tablet_manager.cc:1367] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.007s
I20240502 01:49:55.007375 18433 ts_tablet_manager.cc:1367] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.015s
I20240502 01:49:55.008527 18434 tablet_bootstrap.cc:654] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.009936 18435 raft_consensus.cc:348] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.010649 18435 raft_consensus.cc:374] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.010907 18435 raft_consensus.cc:729] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.011498 18435 consensus_queue.cc:260] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.012053 18311 raft_consensus.cc:3037] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.011847 18433 raft_consensus.cc:348] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.012569 18433 raft_consensus.cc:374] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.012862 18433 raft_consensus.cc:729] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.014559 18435 ts_tablet_manager.cc:1398] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:49:55.014135 18433 consensus_queue.cc:260] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.015494 18435 tablet_bootstrap.cc:492] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:55.018579 18433 ts_tablet_manager.cc:1398] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240502 01:49:55.019503 18433 tablet_bootstrap.cc:492] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:55.025468 18434 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.025978 18434 ts_tablet_manager.cc:1367] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.007s
I20240502 01:49:55.026685 18435 tablet_bootstrap.cc:654] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.036465 18434 raft_consensus.cc:348] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.037273 18434 raft_consensus.cc:374] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.037556 18434 raft_consensus.cc:729] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.037853 18608 raft_consensus.cc:504] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.038179 18434 consensus_queue.cc:260] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.040665 18434 ts_tablet_manager.cc:1398] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240502 01:49:55.041321 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd8414d528f4407b7369da924f707c5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:55.041574 18434 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.042100 18433 tablet_bootstrap.cc:654] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.043637 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd8414d528f4407b7369da924f707c5" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:55.044402 18388 raft_consensus.cc:3037] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.041918 18310 raft_consensus.cc:3037] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.048524 18608 leader_election.cc:288] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.058522 18435 tablet_bootstrap.cc:492] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.059140 18435 ts_tablet_manager.cc:1367] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.003s
I20240502 01:49:55.066304 18310 raft_consensus.cc:2454] T cdd8414d528f4407b7369da924f707c5 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:55.067633 18201 leader_election.cc:302] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:55.068634 18608 raft_consensus.cc:2781] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:55.069151 18608 raft_consensus.cc:686] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:55.070026 18608 consensus_queue.cc:237] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.074620 18434 tablet_bootstrap.cc:654] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.076117 18806 raft_consensus.cc:482] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.081868 18388 raft_consensus.cc:2454] T cdd8414d528f4407b7369da924f707c5 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:55.083746 18806 raft_consensus.cc:504] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.086169 18806 leader_election.cc:288] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.087363 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:55.090191 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e60f2654045a39491fd7e358029da" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.121681 18433 tablet_bootstrap.cc:492] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:55.122262 18433 ts_tablet_manager.cc:1367] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.103s user 0.018s sys 0.004s
I20240502 01:49:55.100368 18435 raft_consensus.cc:348] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.110893 18133 catalog_manager.cc:5484] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:55.125828 18435 raft_consensus.cc:374] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.126225 18435 raft_consensus.cc:729] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.126993 18435 consensus_queue.cc:260] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.128593 18435 ts_tablet_manager.cc:1398] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20240502 01:49:55.130157 18435 tablet_bootstrap.cc:492] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:55.126081 18280 leader_election.cc:302] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:55.133471 18806 raft_consensus.cc:2726] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:55.135305 18434 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.135778 18434 ts_tablet_manager.cc:1367] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.094s user 0.010s sys 0.001s
I20240502 01:49:55.124837 18433 raft_consensus.cc:348] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.139149 18433 raft_consensus.cc:374] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.139452 18433 raft_consensus.cc:729] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.140182 18433 consensus_queue.cc:260] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.141482 18434 raft_consensus.cc:348] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.142189 18434 raft_consensus.cc:374] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.142438 18434 raft_consensus.cc:729] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.143019 18434 consensus_queue.cc:260] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.143479 18433 ts_tablet_manager.cc:1398] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.021s user 0.002s sys 0.003s
I20240502 01:49:55.147570 18435 tablet_bootstrap.cc:654] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.150319 18434 ts_tablet_manager.cc:1398] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 01:49:55.155809 18434 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.161834 18309 raft_consensus.cc:3037] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.163802 18434 tablet_bootstrap.cc:654] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.177254 18434 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.177884 18434 ts_tablet_manager.cc:1367] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.009s
I20240502 01:49:55.180810 18434 raft_consensus.cc:348] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.181985 18434 raft_consensus.cc:374] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.182312 18434 raft_consensus.cc:729] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.183151 18434 consensus_queue.cc:260] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.185329 18434 ts_tablet_manager.cc:1398] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:49:55.186077 18678 raft_consensus.cc:482] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.186360 18434 tablet_bootstrap.cc:492] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.189823 18678 raft_consensus.cc:504] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.192327 18678 leader_election.cc:288] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:55.193986 18434 tablet_bootstrap.cc:654] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.194021 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2837217d424de797067c2daede656c" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:55.196441 18356 leader_election.cc:341] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.203812 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2837217d424de797067c2daede656c" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.215219 18725 raft_consensus.cc:482] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.215906 18725 raft_consensus.cc:504] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.218045 18725 leader_election.cc:288] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.223902 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a4278616b44d738b2b7aa7eafef008" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:55.224661 18234 raft_consensus.cc:2454] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:55.224105 18316 raft_consensus.cc:3037] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.231069 18773 raft_consensus.cc:482] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.231715 18773 raft_consensus.cc:504] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.235898 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7210798da7624de29528dfd61b8a1efa" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:55.237057 18773 leader_election.cc:288] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:55.237514 18356 leader_election.cc:341] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.239298 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a4278616b44d738b2b7aa7eafef008" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.242350 18387 raft_consensus.cc:2454] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:55.243228 18280 leader_election.cc:302] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:55.244030 18725 raft_consensus.cc:2781] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:55.244383 18725 raft_consensus.cc:482] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:55.244724 18725 raft_consensus.cc:3037] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:55.239897 18355 leader_election.cc:341] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.252486 18355 leader_election.cc:302] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:55.253165 18773 raft_consensus.cc:2726] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:55.257546 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7210798da7624de29528dfd61b8a1efa" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:55.275236 18355 leader_election.cc:341] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.275689 18355 leader_election.cc:302] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:55.288096 18433 tablet_bootstrap.cc:492] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:55.283622 18738 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 01:49:55.309376 18434 tablet_bootstrap.cc:492] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.309921 18434 ts_tablet_manager.cc:1367] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.124s user 0.048s sys 0.057s
I20240502 01:49:55.310820 18608 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:55.316156 18792 raft_consensus.cc:2726] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:49:55.321584 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 46380.3us
I20240502 01:49:55.328212 18434 raft_consensus.cc:348] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.329391 18434 raft_consensus.cc:374] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.329813 18434 raft_consensus.cc:729] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.330706 18434 consensus_queue.cc:260] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.333912 18434 ts_tablet_manager.cc:1398] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.024s user 0.007s sys 0.003s
I20240502 01:49:55.335311 18434 tablet_bootstrap.cc:492] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.340206 18234 raft_consensus.cc:3037] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.344686 18434 tablet_bootstrap.cc:654] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.362802 18614 raft_consensus.cc:482] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.364041 18614 raft_consensus.cc:504] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.372844 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b5bb05d88c44618ab1fde55d4dd7a2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:55.375697 18356 leader_election.cc:341] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.371618 18614 leader_election.cc:288] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:55.420985 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b5bb05d88c44618ab1fde55d4dd7a2" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.432000 18725 raft_consensus.cc:504] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.434087 18725 leader_election.cc:288] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:55.437839 18355 leader_election.cc:341] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.438325 18355 leader_election.cc:302] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:55.440307 18792 raft_consensus.cc:2726] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:55.446339 18435 tablet_bootstrap.cc:492] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.447268 18433 tablet_bootstrap.cc:654] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.446789 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a4278616b44d738b2b7aa7eafef008" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:55.449677 18392 raft_consensus.cc:3037] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.449694 18435 ts_tablet_manager.cc:1367] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.320s user 0.067s sys 0.156s
W20240502 01:49:55.468756 18203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 46402.3us
I20240502 01:49:55.475641 18392 raft_consensus.cc:2454] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:55.477999 18280 leader_election.cc:302] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:55.480365 18724 raft_consensus.cc:2781] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:55.480572 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a4278616b44d738b2b7aa7eafef008" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:55.481245 18238 raft_consensus.cc:3037] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.490767 18434 tablet_bootstrap.cc:492] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.491375 18434 ts_tablet_manager.cc:1367] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.156s user 0.040s sys 0.043s
I20240502 01:49:55.504218 18434 raft_consensus.cc:348] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.504971 18434 raft_consensus.cc:374] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.505323 18434 raft_consensus.cc:729] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.505954 18434 consensus_queue.cc:260] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.495024 18724 raft_consensus.cc:686] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:55.509119 18724 consensus_queue.cc:237] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.513510 18434 ts_tablet_manager.cc:1398] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20240502 01:49:55.515375 18434 tablet_bootstrap.cc:492] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.516155 18435 raft_consensus.cc:348] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.518998 18435 raft_consensus.cc:374] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.519274 18435 raft_consensus.cc:729] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.519882 18435 consensus_queue.cc:260] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.533239 18435 ts_tablet_manager.cc:1398] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.083s user 0.006s sys 0.000s
I20240502 01:49:55.534302 18435 tablet_bootstrap.cc:492] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:55.538861 18738 raft_consensus.cc:482] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98f3bf374d49481cb6ebe0672e5b81dc)
I20240502 01:49:55.539393 18738 raft_consensus.cc:504] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.532176 18806 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.544373 18806 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.546826 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.543850 18238 raft_consensus.cc:2454] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:55.549082 18131 catalog_manager.cc:5484] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:55.561841 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.562629 18392 raft_consensus.cc:2379] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate d6698cd7a2d740368c0c89233e16f04c in this term.
I20240502 01:49:55.547159 18806 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.564038 18608 raft_consensus.cc:482] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.564545 18608 raft_consensus.cc:504] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.566018 18433 tablet_bootstrap.cc:492] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:55.566471 18433 ts_tablet_manager.cc:1367] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.301s user 0.013s sys 0.009s
I20240502 01:49:55.566515 18608 leader_election.cc:288] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.567337 18434 tablet_bootstrap.cc:654] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.570884 18738 leader_election.cc:288] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.572402 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.572733 18608 consensus_queue.cc:1035] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:55.574848 18433 raft_consensus.cc:348] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.575544 18433 raft_consensus.cc:374] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.575803 18433 raft_consensus.cc:729] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.576928 18433 consensus_queue.cc:260] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.578998 18433 ts_tablet_manager.cc:1398] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
W20240502 01:49:55.580551 18201 leader_election.cc:341] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.581002 18433 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:55.593107 18852 consensus_queue.cc:1035] T cdd8414d528f4407b7369da924f707c5 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:55.594967 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.599431 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:55.600546 18234 raft_consensus.cc:2379] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:55.603653 18280 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:55.610020 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420ebc81e61440d997f99ad838b88f79" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.611547 18203 leader_election.cc:302] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:55.612324 18806 raft_consensus.cc:3037] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.612435 18852 raft_consensus.cc:2726] T 420ebc81e61440d997f99ad838b88f79 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:49:55.627132 18386 raft_consensus.cc:2454] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:55.646548 18434 tablet_bootstrap.cc:492] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.648730 18434 ts_tablet_manager.cc:1367] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.134s user 0.031s sys 0.007s
I20240502 01:49:55.644486 18435 tablet_bootstrap.cc:654] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.649264 18203 leader_election.cc:302] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:55.643757 18433 tablet_bootstrap.cc:654] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.650301 18852 raft_consensus.cc:2781] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:55.653416 18852 raft_consensus.cc:482] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:55.653793 18852 raft_consensus.cc:3037] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.656551 18434 raft_consensus.cc:348] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.662396 18434 raft_consensus.cc:374] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.661695 18806 raft_consensus.cc:2726] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:55.664920 18852 raft_consensus.cc:504] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.673416 18852 leader_election.cc:288] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.674301 18435 tablet_bootstrap.cc:492] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.674117 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:55.674758 18435 ts_tablet_manager.cc:1367] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.010s
I20240502 01:49:55.676669 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f441096d7e74258a6c1629ba45bb1f3" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:55.677299 18386 raft_consensus.cc:3037] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.669512 18434 raft_consensus.cc:729] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.682188 18434 consensus_queue.cc:260] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.684746 18434 ts_tablet_manager.cc:1398] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20240502 01:49:55.684787 18433 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:55.685441 18433 ts_tablet_manager.cc:1367] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.002s
I20240502 01:49:55.685753 18434 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.686484 18311 raft_consensus.cc:3037] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:55.688654 18433 raft_consensus.cc:348] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.689723 18433 raft_consensus.cc:374] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.690060 18433 raft_consensus.cc:729] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.690888 18433 consensus_queue.cc:260] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.692447 18434 tablet_bootstrap.cc:654] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.692514 18433 ts_tablet_manager.cc:1398] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:49:55.694298 18433 tablet_bootstrap.cc:492] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:55.700780 18433 tablet_bootstrap.cc:654] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.706020 18435 raft_consensus.cc:348] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.708264 18435 raft_consensus.cc:374] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.709187 18435 raft_consensus.cc:729] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.710059 18311 raft_consensus.cc:2454] T 7f441096d7e74258a6c1629ba45bb1f3 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:55.714200 18435 consensus_queue.cc:260] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.715906 18433 tablet_bootstrap.cc:492] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:55.716497 18433 ts_tablet_manager.cc:1367] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.002s
I20240502 01:49:55.718951 18201 leader_election.cc:302] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:55.719792 18608 raft_consensus.cc:2781] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:55.719261 18433 raft_consensus.cc:348] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.720332 18608 raft_consensus.cc:686] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:55.720603 18433 raft_consensus.cc:374] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.721077 18433 raft_consensus.cc:729] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.721175 18608 consensus_queue.cc:237] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.721873 18433 consensus_queue.cc:260] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.727569 18433 ts_tablet_manager.cc:1398] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:49:55.728711 18433 tablet_bootstrap.cc:492] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c: Bootstrap starting.
I20240502 01:49:55.736567 18434 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.737125 18434 ts_tablet_manager.cc:1367] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.005s
I20240502 01:49:55.739972 18433 tablet_bootstrap.cc:654] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.751210 18433 tablet_bootstrap.cc:492] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c: No bootstrap required, opened a new log
I20240502 01:49:55.751775 18433 ts_tablet_manager.cc:1367] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.000s
I20240502 01:49:55.752487 18386 raft_consensus.cc:2454] T 7f441096d7e74258a6c1629ba45bb1f3 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:55.753839 18434 raft_consensus.cc:348] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.754598 18434 raft_consensus.cc:374] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.754264 18433 raft_consensus.cc:348] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.754877 18434 raft_consensus.cc:729] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.755260 18433 raft_consensus.cc:374] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.755641 18433 raft_consensus.cc:729] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.755748 18434 consensus_queue.cc:260] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.756541 18433 consensus_queue.cc:260] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.758654 18433 ts_tablet_manager.cc:1398] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:49:55.754861 18435 ts_tablet_manager.cc:1398] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.080s user 0.006s sys 0.002s
I20240502 01:49:55.759778 18435 tablet_bootstrap.cc:492] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:55.765936 18435 tablet_bootstrap.cc:654] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.766377 18434 ts_tablet_manager.cc:1398] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20240502 01:49:55.767899 18434 tablet_bootstrap.cc:492] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.774083 18434 tablet_bootstrap.cc:654] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.778860 18435 tablet_bootstrap.cc:492] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.779502 18435 ts_tablet_manager.cc:1367] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240502 01:49:55.782308 18435 raft_consensus.cc:348] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.785830 18435 raft_consensus.cc:374] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.786228 18435 raft_consensus.cc:729] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.786962 18435 consensus_queue.cc:260] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.791361 18435 ts_tablet_manager.cc:1398] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:49:55.792567 18435 tablet_bootstrap.cc:492] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:55.798532 18435 tablet_bootstrap.cc:654] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.803609 18434 tablet_bootstrap.cc:492] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.804245 18434 ts_tablet_manager.cc:1367] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.011s
I20240502 01:49:55.805317 18435 tablet_bootstrap.cc:492] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.806010 18435 ts_tablet_manager.cc:1367] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240502 01:49:55.806802 18434 raft_consensus.cc:348] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.807570 18434 raft_consensus.cc:374] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.807849 18434 raft_consensus.cc:729] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.845321 18434 consensus_queue.cc:260] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.845603 18435 raft_consensus.cc:348] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.847326 18435 raft_consensus.cc:374] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.847884 18435 raft_consensus.cc:729] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.849512 18435 consensus_queue.cc:260] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.856565 18851 raft_consensus.cc:482] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.857796 18851 raft_consensus.cc:504] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.861153 18435 ts_tablet_manager.cc:1398] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.055s user 0.004s sys 0.007s
I20240502 01:49:55.867905 18435 tablet_bootstrap.cc:492] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:55.887966 18434 ts_tablet_manager.cc:1398] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.083s user 0.009s sys 0.003s
I20240502 01:49:55.888227 18738 raft_consensus.cc:482] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.888970 18738 raft_consensus.cc:504] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.895844 18738 leader_election.cc:288] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.889925 18129 catalog_manager.cc:5484] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:55.897090 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
W20240502 01:49:55.899618 18201 leader_election.cc:341] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.889920 18434 tablet_bootstrap.cc:492] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:55.904335 18723 raft_consensus.cc:482] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.904522 18435 tablet_bootstrap.cc:654] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.905581 18723 raft_consensus.cc:504] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.914458 18434 tablet_bootstrap.cc:654] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:55.914150 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.919577 18851 leader_election.cc:288] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.921869 18792 raft_consensus.cc:482] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.932471 18792 raft_consensus.cc:504] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.933624 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2318233df2844c48918882f56859c152" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:55.933825 18390 raft_consensus.cc:2454] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:55.938959 18614 raft_consensus.cc:482] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 01:49:55.939656 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 60591.6us
I20240502 01:49:55.939705 18614 raft_consensus.cc:504] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.947623 18723 leader_election.cc:288] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.958066 18434 tablet_bootstrap.cc:492] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:55.958663 18434 ts_tablet_manager.cc:1367] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.069s user 0.021s sys 0.022s
I20240502 01:49:55.960984 18614 leader_election.cc:288] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
W20240502 01:49:55.961858 18280 leader_election.cc:341] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.963644 18792 leader_election.cc:288] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:55.964591 18847 raft_consensus.cc:482] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:55.981696 18847 raft_consensus.cc:504] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.967350 18203 leader_election.cc:302] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:55.971560 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2318233df2844c48918882f56859c152" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:55.972306 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6049ac47c1fb40289b45cadccb8ecf97" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.972697 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.980719 18434 raft_consensus.cc:348] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:55.985764 18392 raft_consensus.cc:2454] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:55.986393 18434 raft_consensus.cc:374] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:55.986905 18434 raft_consensus.cc:729] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:55.989831 18847 leader_election.cc:288] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:55.990042 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2837217d424de797067c2daede656c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:55.991672 18608 raft_consensus.cc:2781] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:55.988974 18434 consensus_queue.cc:260] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:55.992709 18201 leader_election.cc:341] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:55.995374 18435 tablet_bootstrap.cc:492] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:55.996855 18435 ts_tablet_manager.cc:1367] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.130s user 0.004s sys 0.012s
I20240502 01:49:55.993986 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b5bb05d88c44618ab1fde55d4dd7a2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:55.996397 18316 raft_consensus.cc:2454] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:55.999466 18238 raft_consensus.cc:2454] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:56.001806 18434 ts_tablet_manager.cc:1398] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20240502 01:49:55.993357 18608 raft_consensus.cc:482] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.002732 18608 raft_consensus.cc:3037] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.003122 18434 tablet_bootstrap.cc:492] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:56.004999 18435 raft_consensus.cc:348] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.010555 18435 raft_consensus.cc:374] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:56.010890 18435 raft_consensus.cc:729] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
W20240502 01:49:56.013713 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 44016.9us
I20240502 01:49:56.013650 18435 consensus_queue.cc:260] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.016024 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6049ac47c1fb40289b45cadccb8ecf97" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:56.016755 18237 raft_consensus.cc:2454] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 0.
I20240502 01:49:56.020076 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:56.024477 18435 ts_tablet_manager.cc:1398] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.027s user 0.000s sys 0.009s
I20240502 01:49:56.019379 18608 raft_consensus.cc:504] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.028493 18876 raft_consensus.cc:482] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:56.042742 18231 raft_consensus.cc:2379] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6698cd7a2d740368c0c89233e16f04c in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:56.042441 18876 raft_consensus.cc:504] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.029250 18435 tablet_bootstrap.cc:492] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:56.029855 18280 leader_election.cc:302] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:56.032956 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2837217d424de797067c2daede656c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:56.044553 18725 raft_consensus.cc:2781] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:56.044667 18392 raft_consensus.cc:2454] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:56.044968 18725 raft_consensus.cc:482] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.045337 18725 raft_consensus.cc:3037] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.045663 18851 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:56.046380 18203 leader_election.cc:302] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:56.034341 18434 tablet_bootstrap.cc:654] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:56.036659 18355 leader_election.cc:341] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:56.048487 18847 raft_consensus.cc:2781] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:56.049872 18847 raft_consensus.cc:482] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.050254 18847 raft_consensus.cc:3037] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.048656 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084a332369434663a3a306d0fb57e927" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:56.056268 18231 raft_consensus.cc:2454] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:56.048868 18876 leader_election.cc:288] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.060338 18280 leader_election.cc:302] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:56.060741 18608 leader_election.cc:288] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.063158 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:56.063745 18392 raft_consensus.cc:3037] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.064369 18876 raft_consensus.cc:2781] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:56.064770 18876 raft_consensus.cc:482] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.065094 18876 raft_consensus.cc:3037] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.072988 18392 raft_consensus.cc:2454] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:56.060429 18316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:56.075277 18203 leader_election.cc:302] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:56.077554 18435 tablet_bootstrap.cc:654] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:56.077730 18201 leader_election.cc:341] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:56.080192 18356 leader_election.cc:302] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:56.085911 18847 raft_consensus.cc:504] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.079922 18725 raft_consensus.cc:504] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.087303 18874 raft_consensus.cc:2781] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:56.087824 18874 raft_consensus.cc:482] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.088286 18874 raft_consensus.cc:3037] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.089226 18725 leader_election.cc:288] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.091087 18847 leader_election.cc:288] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.091940 18280 leader_election.cc:302] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:56.092737 18725 raft_consensus.cc:2781] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:56.093161 18725 raft_consensus.cc:482] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.093582 18725 raft_consensus.cc:3037] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.095096 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2837217d424de797067c2daede656c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:56.095188 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2837217d424de797067c2daede656c" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:56.096011 18392 raft_consensus.cc:3037] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.098212 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b5bb05d88c44618ab1fde55d4dd7a2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:56.099046 18390 raft_consensus.cc:2454] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:56.093748 18776 raft_consensus.cc:2781] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:56.100328 18776 raft_consensus.cc:686] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
W20240502 01:49:56.101202 18201 leader_election.cc:341] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:56.101300 18776 consensus_queue.cc:237] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.101986 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084a332369434663a3a306d0fb57e927" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:56.105274 18391 raft_consensus.cc:2454] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 0.
I20240502 01:49:56.109242 18874 raft_consensus.cc:504] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.111269 18874 leader_election.cc:288] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:56.107765 18392 raft_consensus.cc:2454] T cf2837217d424de797067c2daede656c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:56.111541 18725 raft_consensus.cc:504] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.113473 18203 leader_election.cc:302] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:56.114634 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b5bb05d88c44618ab1fde55d4dd7a2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:56.115388 18231 raft_consensus.cc:3037] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.116600 18608 raft_consensus.cc:2781] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:56.116771 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b5bb05d88c44618ab1fde55d4dd7a2" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:56.117449 18392 raft_consensus.cc:3037] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.119009 18608 raft_consensus.cc:686] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:56.117971 18356 leader_election.cc:302] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:56.123977 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6049ac47c1fb40289b45cadccb8ecf97" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:56.124668 18237 raft_consensus.cc:3037] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.124893 18874 raft_consensus.cc:482] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:56.125351 18608 consensus_queue.cc:237] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.124780 18879 raft_consensus.cc:2726] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:56.134406 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6049ac47c1fb40289b45cadccb8ecf97" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:56.135072 18311 raft_consensus.cc:3037] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.125453 18874 raft_consensus.cc:504] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.144251 18874 leader_election.cc:288] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:49:56.145443 18806 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:56.152218 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:56.150899 18725 leader_election.cc:288] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.155094 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2318233df2844c48918882f56859c152" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
W20240502 01:49:56.155452 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:56.155699 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2318233df2844c48918882f56859c152" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:56.156402 18385 raft_consensus.cc:3037] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:49:56.160297 18280 leader_election.cc:341] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:56.164556 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "d6698cd7a2d740368c0c89233e16f04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
W20240502 01:49:56.168515 18355 leader_election.cc:341] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:49:56.170305 18356 leader_election.cc:341] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:56.170724 18356 leader_election.cc:302] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:56.172236 18874 raft_consensus.cc:2726] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:56.179073 18392 raft_consensus.cc:2454] T 09b5bb05d88c44618ab1fde55d4dd7a2 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:56.186431 18280 leader_election.cc:302] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:56.180452 18231 raft_consensus.cc:2454] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:56.187526 18806 raft_consensus.cc:2781] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:56.190799 18806 raft_consensus.cc:686] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:56.191718 18806 consensus_queue.cc:237] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.195941 18237 raft_consensus.cc:2454] T 6049ac47c1fb40289b45cadccb8ecf97 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:56.209371 18876 raft_consensus.cc:504] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.213760 18356 leader_election.cc:302] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters:
I20240502 01:49:56.214543 18876 leader_election.cc:288] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Requested vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.214686 18874 raft_consensus.cc:2781] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:56.210970 18385 raft_consensus.cc:2454] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:56.215173 18874 raft_consensus.cc:686] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Becoming Leader. State: Replica: d6698cd7a2d740368c0c89233e16f04c, State: Running, Role: LEADER
I20240502 01:49:56.216167 18874 consensus_queue.cc:237] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.220006 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084a332369434663a3a306d0fb57e927" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794"
I20240502 01:49:56.226086 18238 raft_consensus.cc:3037] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.221028 18280 leader_election.cc:302] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c; no voters: 26c26643de084d6b8acaff73fa67f794
I20240502 01:49:56.229166 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084a332369434663a3a306d0fb57e927" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:56.229894 18391 raft_consensus.cc:3037] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.231698 18725 raft_consensus.cc:2781] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:56.232491 18725 raft_consensus.cc:686] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:56.233429 18725 consensus_queue.cc:237] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.237216 18435 tablet_bootstrap.cc:492] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:56.237736 18435 ts_tablet_manager.cc:1367] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.209s user 0.022s sys 0.012s
I20240502 01:49:56.240743 18435 raft_consensus.cc:348] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.241500 18435 raft_consensus.cc:374] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:56.241802 18435 raft_consensus.cc:729] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:56.245525 18434 tablet_bootstrap.cc:492] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:56.246058 18434 ts_tablet_manager.cc:1367] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.243s user 0.030s sys 0.011s
I20240502 01:49:56.249915 18435 consensus_queue.cc:260] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.248999 18738 consensus_queue.cc:1035] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 01:49:56.254554 18238 raft_consensus.cc:2454] T 084a332369434663a3a306d0fb57e927 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:56.263060 18280 leader_election.cc:302] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:56.266088 18875 raft_consensus.cc:2781] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:56.265863 18435 ts_tablet_manager.cc:1398] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.028s user 0.006s sys 0.002s
I20240502 01:49:56.262588 18434 raft_consensus.cc:348] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.267730 18434 raft_consensus.cc:374] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:56.256181 18311 raft_consensus.cc:2454] T 6049ac47c1fb40289b45cadccb8ecf97 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6698cd7a2d740368c0c89233e16f04c in term 1.
I20240502 01:49:56.268064 18434 raft_consensus.cc:729] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:56.276615 18434 consensus_queue.cc:260] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.273406 18875 raft_consensus.cc:686] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Becoming Leader. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Running, Role: LEADER
I20240502 01:49:56.339622 18852 raft_consensus.cc:482] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:56.340312 18852 raft_consensus.cc:504] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.434129 18391 raft_consensus.cc:2454] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f3bf374d49481cb6ebe0672e5b81dc in term 1.
I20240502 01:49:56.435331 18434 ts_tablet_manager.cc:1398] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.189s user 0.014s sys 0.004s
I20240502 01:49:56.438016 18435 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:56.439522 18852 leader_election.cc:288] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
W20240502 01:49:56.443303 18355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 159552us
I20240502 01:49:56.446434 18875 consensus_queue.cc:237] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
W20240502 01:49:56.451304 18201 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.130:33629, user_credentials={real_user=slave}} blocked reactor thread for 152072us
I20240502 01:49:56.467630 18608 consensus_queue.cc:1035] T 7f441096d7e74258a6c1629ba45bb1f3 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:56.481156 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:56.490665 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:56.491508 18387 raft_consensus.cc:2454] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
W20240502 01:49:56.493330 18201 leader_election.cc:341] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:56.557366 18435 tablet_bootstrap.cc:654] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:56.730792 18435 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:56.731280 18435 ts_tablet_manager.cc:1367] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.294s user 0.015s sys 0.000s
I20240502 01:49:56.733661 18435 raft_consensus.cc:348] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.734197 18435 raft_consensus.cc:374] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:56.734442 18435 raft_consensus.cc:729] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:56.734972 18435 consensus_queue.cc:260] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.736968 18435 ts_tablet_manager.cc:1398] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240502 01:49:56.738003 18435 tablet_bootstrap.cc:492] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794: Bootstrap starting.
I20240502 01:49:56.767676 18876 raft_consensus.cc:482] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:56.792863 18876 raft_consensus.cc:504] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.795176 18876 leader_election.cc:288] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.899282 18435 tablet_bootstrap.cc:654] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:56.923434 18895 log.cc:924] Time spent T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: Append to log took a long time: real 0.405s user 0.004s sys 0.009s
I20240502 01:49:56.928046 18435 tablet_bootstrap.cc:492] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794: No bootstrap required, opened a new log
I20240502 01:49:56.928556 18435 ts_tablet_manager.cc:1367] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.006s
I20240502 01:49:56.931586 18203 leader_election.cc:302] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:56.931260 18435 raft_consensus.cc:348] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.932911 18435 raft_consensus.cc:374] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:56.933223 18435 raft_consensus.cc:729] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Initialized, Role: FOLLOWER
I20240502 01:49:56.933871 18435 consensus_queue.cc:260] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.934370 18852 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:56.934875 18852 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:56.936080 18435 ts_tablet_manager.cc:1398] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240502 01:49:56.936151 18891 raft_consensus.cc:2781] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:56.936581 18891 raft_consensus.cc:482] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:56.936861 18891 raft_consensus.cc:3037] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:56.950901 18133 catalog_manager.cc:5484] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:56.952701 18133 catalog_manager.cc:5484] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:56.961747 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:56.962599 18391 raft_consensus.cc:2379] T 0c9cb5b8b9404935a860997c60c29470 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:56.967657 18852 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:56.973203 18434 tablet_bootstrap.cc:492] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:56.980676 18434 tablet_bootstrap.cc:654] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
W20240502 01:49:56.989816 18203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.165.131:45775, user_credentials={real_user=slave}} blocked reactor thread for 57173.9us
I20240502 01:49:56.995191 18724 consensus_queue.cc:1035] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:56.999258 18434 tablet_bootstrap.cc:492] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:56.999724 18434 ts_tablet_manager.cc:1367] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.011s
I20240502 01:49:57.000450 18608 consensus_queue.cc:1035] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.001197 18723 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.003451 18891 raft_consensus.cc:504] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.004776 18434 raft_consensus.cc:348] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.006417 18434 raft_consensus.cc:374] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:57.007565 18434 raft_consensus.cc:729] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:57.008816 18434 consensus_queue.cc:260] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.011010 18434 ts_tablet_manager.cc:1398] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.011s user 0.003s sys 0.001s
I20240502 01:49:57.011977 18434 tablet_bootstrap.cc:492] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:57.014413 18311 raft_consensus.cc:3037] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.014973 18310 raft_consensus.cc:3037] T cf2837217d424de797067c2daede656c P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.023983 18891 leader_election.cc:288] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:57.024555 18238 raft_consensus.cc:3037] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.030884 18792 consensus_queue.cc:1035] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.032420 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9cb5b8b9404935a860997c60c29470" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:57.044265 18434 tablet_bootstrap.cc:654] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:57.050787 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:57.051537 18310 raft_consensus.cc:2454] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:57.053584 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:57.054225 18392 raft_consensus.cc:2454] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:57.058569 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:57.065105 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:57.065791 18385 raft_consensus.cc:3037] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.068055 18776 consensus_queue.cc:1035] T 0c9cb5b8b9404935a860997c60c29470 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.071985 18776 consensus_queue.cc:1035] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.075763 18385 raft_consensus.cc:2454] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:57.087958 18896 consensus_queue.cc:1035] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:57.089687 18201 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:57.090986 18738 consensus_queue.cc:1035] T cf2837217d424de797067c2daede656c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.092376 18738 raft_consensus.cc:2781] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:49:57.092298 18280 leader_election.cc:302] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:57.092816 18434 tablet_bootstrap.cc:492] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:57.092844 18738 raft_consensus.cc:482] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:57.093438 18434 ts_tablet_manager.cc:1367] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.082s user 0.020s sys 0.022s
I20240502 01:49:57.093411 18738 raft_consensus.cc:3037] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:49:57.097339 18201 leader_election.cc:341] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:49:57.101716 18738 raft_consensus.cc:504] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.105253 18203 leader_election.cc:302] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c; no voters: 98f3bf374d49481cb6ebe0672e5b81dc
I20240502 01:49:57.106882 18898 raft_consensus.cc:2781] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:57.107331 18898 raft_consensus.cc:686] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:57.108134 18898 consensus_queue.cc:237] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.109918 18738 leader_election.cc:288] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:57.110090 18131 catalog_manager.cc:5484] T 09b5bb05d88c44618ab1fde55d4dd7a2 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:57.110699 18874 consensus_queue.cc:1035] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.110956 18896 raft_consensus.cc:2726] T 0c9cb5b8b9404935a860997c60c29470 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:57.114648 18434 raft_consensus.cc:348] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.116128 18434 raft_consensus.cc:374] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:57.116453 18434 raft_consensus.cc:729] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:57.117101 18434 consensus_queue.cc:260] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.120483 18434 ts_tablet_manager.cc:1398] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20240502 01:49:57.122395 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:57.123004 18309 raft_consensus.cc:3037] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:57.123174 18434 tablet_bootstrap.cc:492] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc: Bootstrap starting.
I20240502 01:49:57.126499 18897 consensus_queue.cc:1035] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:57.133854 18309 raft_consensus.cc:2454] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 2.
I20240502 01:49:57.136509 18201 leader_election.cc:302] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:57.136709 18897 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.137874 18891 raft_consensus.cc:2781] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:49:57.138373 18891 raft_consensus.cc:686] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [term 2 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:57.139219 18891 consensus_queue.cc:237] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.143301 18434 tablet_bootstrap.cc:654] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:57.151746 18131 catalog_manager.cc:5484] T 6049ac47c1fb40289b45cadccb8ecf97 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6698cd7a2d740368c0c89233e16f04c (127.16.165.131). New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } }
I20240502 01:49:57.157387 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b1b170e56d4e1591f3991e0883ce16" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:57.158057 18897 consensus_queue.cc:1035] T 084a332369434663a3a306d0fb57e927 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.158049 18385 raft_consensus.cc:3037] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:49:57.169541 18891 raft_consensus.cc:482] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:57.170102 18891 raft_consensus.cc:504] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.172551 18891 leader_election.cc:288] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:57.173344 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2318233df2844c48918882f56859c152" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:57.170032 18131 catalog_manager.cc:5484] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } }
I20240502 01:49:57.175967 18385 raft_consensus.cc:2454] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 2.
I20240502 01:49:57.197856 18434 tablet_bootstrap.cc:492] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc: No bootstrap required, opened a new log
I20240502 01:49:57.198501 18434 ts_tablet_manager.cc:1367] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.000s
I20240502 01:49:57.201234 18434 raft_consensus.cc:348] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.202294 18434 raft_consensus.cc:374] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:57.203127 18434 raft_consensus.cc:729] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f3bf374d49481cb6ebe0672e5b81dc, State: Initialized, Role: FOLLOWER
I20240502 01:49:57.208060 18434 consensus_queue.cc:260] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.211032 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2318233df2844c48918882f56859c152" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:57.215950 18434 ts_tablet_manager.cc:1398] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240502 01:49:57.219405 18776 raft_consensus.cc:482] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:57.219940 18776 raft_consensus.cc:504] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.221823 18776 leader_election.cc:288] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:57.224596 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7210798da7624de29528dfd61b8a1efa" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" is_pre_election: true
I20240502 01:49:57.225318 18310 raft_consensus.cc:2454] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:57.226336 18201 leader_election.cc:302] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:57.227020 18776 raft_consensus.cc:2781] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:57.227425 18776 raft_consensus.cc:482] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:57.227819 18776 raft_consensus.cc:3037] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.229247 18131 catalog_manager.cc:5484] T 084a332369434663a3a306d0fb57e927 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130). New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } }
W20240502 01:49:57.237160 18183 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:49:57.240641 18203 leader_election.cc:302] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794; no voters: 98f3bf374d49481cb6ebe0672e5b81dc, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:57.241605 18738 raft_consensus.cc:2726] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:49:57.248497 18776 raft_consensus.cc:504] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.249270 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7210798da7624de29528dfd61b8a1efa" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:57.249935 18392 raft_consensus.cc:2454] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 0.
I20240502 01:49:57.250768 18776 leader_election.cc:288] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Requested vote from peers 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:57.252389 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7210798da7624de29528dfd61b8a1efa" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f3bf374d49481cb6ebe0672e5b81dc"
I20240502 01:49:57.252983 18310 raft_consensus.cc:3037] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.260460 18310 raft_consensus.cc:2454] T 7210798da7624de29528dfd61b8a1efa P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:57.261299 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7210798da7624de29528dfd61b8a1efa" candidate_uuid: "26c26643de084d6b8acaff73fa67f794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c"
I20240502 01:49:57.261963 18385 raft_consensus.cc:3037] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.261991 18201 leader_election.cc:302] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c26643de084d6b8acaff73fa67f794, 98f3bf374d49481cb6ebe0672e5b81dc; no voters:
I20240502 01:49:57.264695 18898 raft_consensus.cc:2781] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:57.265162 18898 raft_consensus.cc:686] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Becoming Leader. State: Replica: 26c26643de084d6b8acaff73fa67f794, State: Running, Role: LEADER
I20240502 01:49:57.265913 18898 consensus_queue.cc:237] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.273497 18385 raft_consensus.cc:2454] T 7210798da7624de29528dfd61b8a1efa P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c26643de084d6b8acaff73fa67f794 in term 1.
I20240502 01:49:57.297756 18131 catalog_manager.cc:5484] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:57.299418 18131 catalog_manager.cc:5484] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:57.301067 18131 catalog_manager.cc:5484] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: term changed from 0 to 2, leader changed from <none> to 26c26643de084d6b8acaff73fa67f794 (127.16.165.129). New cstate: current_term: 2 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:57.421470 18851 raft_consensus.cc:482] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:57.421850 18851 raft_consensus.cc:504] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } }
I20240502 01:49:57.424155 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6698cd7a2d740368c0c89233e16f04c" is_pre_election: true
I20240502 01:49:57.425004 18385 raft_consensus.cc:2379] T 4c634b7903104fb1aebaa8b7722fd417 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98f3bf374d49481cb6ebe0672e5b81dc in current term 1: Already voted for candidate 26c26643de084d6b8acaff73fa67f794 in this term.
I20240502 01:49:57.425056 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c634b7903104fb1aebaa8b7722fd417" candidate_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c26643de084d6b8acaff73fa67f794" is_pre_election: true
I20240502 01:49:57.428293 18280 leader_election.cc:302] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98f3bf374d49481cb6ebe0672e5b81dc; no voters: 26c26643de084d6b8acaff73fa67f794, d6698cd7a2d740368c0c89233e16f04c
I20240502 01:49:57.429014 18876 raft_consensus.cc:3037] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:57.423269 18851 leader_election.cc:288] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913), d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:49:57.434847 18876 raft_consensus.cc:2726] T 4c634b7903104fb1aebaa8b7722fd417 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:49:57.521689 18310 raft_consensus.cc:1261] T 22b1b170e56d4e1591f3991e0883ce16 P 98f3bf374d49481cb6ebe0672e5b81dc [term 2 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:49:57.524526 18891 consensus_queue.cc:1035] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.540064 18385 raft_consensus.cc:1261] T 22b1b170e56d4e1591f3991e0883ce16 P d6698cd7a2d740368c0c89233e16f04c [term 2 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:49:57.541821 18852 consensus_queue.cc:1035] T 22b1b170e56d4e1591f3991e0883ce16 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.556821 18891 consensus_queue.cc:1035] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:49:57.568785 18165 debug-util.cc:398] Leaking SignalData structure 0x7b080008eaa0 after lost signal to thread 17049
W20240502 01:49:57.569825 18165 debug-util.cc:398] Leaking SignalData structure 0x7b0800064180 after lost signal to thread 18169
W20240502 01:49:57.570924 18165 debug-util.cc:398] Leaking SignalData structure 0x7b08002ab4c0 after lost signal to thread 18264
I20240502 01:49:57.572196 18898 consensus_queue.cc:1035] T 4c634b7903104fb1aebaa8b7722fd417 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:49:57.579157 18165 debug-util.cc:398] Leaking SignalData structure 0x7b08001ae8e0 after lost signal to thread 18341
W20240502 01:49:57.580255 18165 debug-util.cc:398] Leaking SignalData structure 0x7b08001749c0 after lost signal to thread 18417
I20240502 01:49:57.769696 18776 consensus_queue.cc:1035] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:57.781675 18898 consensus_queue.cc:1035] T 7210798da7624de29528dfd61b8a1efa P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:59.242708 17046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:59.250844 18942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:59.251369 18943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:59.254273 18945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:59.255674 17046 server_base.cc:1034] running on GCE node
I20240502 01:49:59.257112 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:59.257413 17046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:59.257650 17046 hybrid_clock.cc:648] HybridClock initialized: now 1714614599257626 us; error 0 us; skew 500 ppm
I20240502 01:49:59.258450 17046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:59.264271 17046 webserver.cc:460] Webserver started at http://127.16.165.132:33723/ using document root <none> and password file <none>
I20240502 01:49:59.264945 17046 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:59.265280 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:59.265671 17046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:59.267349 17046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-3-root/instance:
uuid: "542154990ef549669640e892eb3292e7"
format_stamp: "Formatted at 2024-05-02 01:49:59 on dist-test-slave-kj4b"
I20240502 01:49:59.275206 17046 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240502 01:49:59.279806 18950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:59.281113 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:49:59.281419 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-3-root
uuid: "542154990ef549669640e892eb3292e7"
format_stamp: "Formatted at 2024-05-02 01:49:59 on dist-test-slave-kj4b"
I20240502 01:49:59.281723 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:59.302304 17046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:59.304159 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:59.306160 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:59.309935 17046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:59.310225 17046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:59.310536 17046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:59.310770 17046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:59.528450 17046 rpc_server.cc:307] RPC server started. Bound to: 127.16.165.132:45743
I20240502 01:49:59.546182 17046 auto_leader_rebalancer-test.cc:405] add a tserver: 542154990ef549669640e892eb3292e7
I20240502 01:49:59.613199 19016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.165.132:45743 every 8 connection(s)
I20240502 01:49:59.666669 19017 heartbeater.cc:347] Connected to a master server at 127.16.165.190:35883
I20240502 01:49:59.667193 19017 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:59.668157 19017 heartbeater.cc:511] Master 127.16.165.190:35883 requested a full tablet report, sending...
I20240502 01:49:59.670569 18129 ts_manager.cc:194] Registered new tserver with Master: 542154990ef549669640e892eb3292e7 (127.16.165.132:45743)
I20240502 01:49:59.672380 18129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56606
I20240502 01:50:00.551594 18311 consensus_queue.cc:237] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:00.559548 18388 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.560765 18237 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.560834 18960 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:00.562758 18960 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:00.585776 18960 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.592923 18392 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.597654 18237 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
W20240502 01:50:00.600916 18281 consensus_peers.cc:485] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3a4278616b44d738b2b7aa7eafef008. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:00.615170 18311 consensus_queue.cc:237] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:00.619589 18129 catalog_manager.cc:5484] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
W20240502 01:50:00.637375 18281 consensus_peers.cc:485] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3eec00736204e3fa0dcf0797f7b0d75. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:00.638283 18391 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.640481 19024 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:00.644138 18238 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.650862 18723 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:00.672763 19024 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.675375 19017 heartbeater.cc:503] Master 127.16.165.190:35883 was elected leader, sending a full tablet report...
I20240502 01:50:00.694702 18237 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.723263 18391 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.778260 18237 consensus_queue.cc:237] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:00.814316 18388 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.815977 18311 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.817358 19022 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:00.819216 19022 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:00.829981 18131 catalog_manager.cc:5484] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.845966 19044 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.847766 18391 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.859836 18311 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.867445 18129 catalog_manager.cc:5484] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
W20240502 01:50:00.885532 18204 consensus_peers.cc:485] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5e09bfc9fdb46af9dd702b4c831e287. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:00.887367 18311 consensus_queue.cc:237] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
W20240502 01:50:00.907847 18281 consensus_peers.cc:485] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5090d855c6864a3d904fe1141d503d9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:00.908176 18391 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.909567 18234 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:00.909596 19050 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:00.910944 18723 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:00.930567 18234 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.931257 18388 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.929102 18723 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:00.957067 18129 catalog_manager.cc:5484] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:01.035851 19054 ts_tablet_manager.cc:902] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:50:01.043792 19054 tablet_copy_client.cc:308] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.130:33629
I20240502 01:50:01.058347 18326 tablet_copy_service.cc:140] P 98f3bf374d49481cb6ebe0672e5b81dc: Received BeginTabletCopySession request for tablet a3a4278616b44d738b2b7aa7eafef008 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:40094)
I20240502 01:50:01.059003 18326 tablet_copy_service.cc:161] P 98f3bf374d49481cb6ebe0672e5b81dc: Beginning new tablet copy session on tablet a3a4278616b44d738b2b7aa7eafef008 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:40094: session id = 542154990ef549669640e892eb3292e7-a3a4278616b44d738b2b7aa7eafef008
I20240502 01:50:01.065917 18326 tablet_copy_source_session.cc:205] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:01.069942 19054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a4278616b44d738b2b7aa7eafef008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:01.090420 19054 tablet_copy_client.cc:783] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:01.091224 19054 tablet_copy_client.cc:647] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:01.099817 19054 tablet_copy_client.cc:523] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:01.108583 19054 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:01.162750 19054 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:01.163628 19054 tablet_bootstrap.cc:492] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:01.164309 19054 ts_tablet_manager.cc:1367] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.056s user 0.033s sys 0.007s
I20240502 01:50:01.167066 19054 raft_consensus.cc:348] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.167943 19054 raft_consensus.cc:729] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:01.168620 19054 consensus_queue.cc:260] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.172626 19054 ts_tablet_manager.cc:1398] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:50:01.182651 18326 tablet_copy_service.cc:342] P 98f3bf374d49481cb6ebe0672e5b81dc: Request end of tablet copy session 542154990ef549669640e892eb3292e7-a3a4278616b44d738b2b7aa7eafef008 received from {username='slave'} at 127.0.0.1:40094
I20240502 01:50:01.183145 18326 tablet_copy_service.cc:433] P 98f3bf374d49481cb6ebe0672e5b81dc: ending tablet copy session 542154990ef549669640e892eb3292e7-a3a4278616b44d738b2b7aa7eafef008 on tablet a3a4278616b44d738b2b7aa7eafef008 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:01.194944 19054 ts_tablet_manager.cc:902] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:50:01.204207 19054 tablet_copy_client.cc:308] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.130:33629
I20240502 01:50:01.205803 18326 tablet_copy_service.cc:140] P 98f3bf374d49481cb6ebe0672e5b81dc: Received BeginTabletCopySession request for tablet c3eec00736204e3fa0dcf0797f7b0d75 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:40094)
I20240502 01:50:01.206357 18326 tablet_copy_service.cc:161] P 98f3bf374d49481cb6ebe0672e5b81dc: Beginning new tablet copy session on tablet c3eec00736204e3fa0dcf0797f7b0d75 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:40094: session id = 542154990ef549669640e892eb3292e7-c3eec00736204e3fa0dcf0797f7b0d75
I20240502 01:50:01.213913 18326 tablet_copy_source_session.cc:205] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:01.217917 19054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3eec00736204e3fa0dcf0797f7b0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:01.233857 19054 tablet_copy_client.cc:783] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:01.234421 19054 tablet_copy_client.cc:647] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:01.240639 19054 tablet_copy_client.cc:523] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:01.260710 19054 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:01.289646 19054 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:01.290580 19054 tablet_bootstrap.cc:492] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:01.291306 19054 ts_tablet_manager.cc:1367] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.002s
I20240502 01:50:01.307639 19054 raft_consensus.cc:348] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.308574 19054 raft_consensus.cc:729] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:01.309223 19054 consensus_queue.cc:260] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.311370 19054 ts_tablet_manager.cc:1398] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.020s user 0.000s sys 0.010s
I20240502 01:50:01.312724 19067 ts_tablet_manager.cc:902] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:50:01.314981 18326 tablet_copy_service.cc:342] P 98f3bf374d49481cb6ebe0672e5b81dc: Request end of tablet copy session 542154990ef549669640e892eb3292e7-c3eec00736204e3fa0dcf0797f7b0d75 received from {username='slave'} at 127.0.0.1:40094
I20240502 01:50:01.315404 18326 tablet_copy_service.cc:433] P 98f3bf374d49481cb6ebe0672e5b81dc: ending tablet copy session 542154990ef549669640e892eb3292e7-c3eec00736204e3fa0dcf0797f7b0d75 on tablet c3eec00736204e3fa0dcf0797f7b0d75 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:01.358952 19067 tablet_copy_client.cc:308] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.129:38913
I20240502 01:50:01.375653 18250 tablet_copy_service.cc:140] P 26c26643de084d6b8acaff73fa67f794: Received BeginTabletCopySession request for tablet d5e09bfc9fdb46af9dd702b4c831e287 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:46486)
I20240502 01:50:01.376302 18250 tablet_copy_service.cc:161] P 26c26643de084d6b8acaff73fa67f794: Beginning new tablet copy session on tablet d5e09bfc9fdb46af9dd702b4c831e287 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:46486: session id = 542154990ef549669640e892eb3292e7-d5e09bfc9fdb46af9dd702b4c831e287
I20240502 01:50:01.383200 18250 tablet_copy_source_session.cc:205] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:01.387748 19067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5e09bfc9fdb46af9dd702b4c831e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:01.402622 19067 tablet_copy_client.cc:783] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:01.403245 19067 tablet_copy_client.cc:647] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:01.410498 19067 tablet_copy_client.cc:523] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:01.424446 19067 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:01.451804 19067 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:01.452683 19067 tablet_bootstrap.cc:492] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:01.453220 19067 ts_tablet_manager.cc:1367] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.015s
I20240502 01:50:01.456038 19067 raft_consensus.cc:348] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.456899 19067 raft_consensus.cc:729] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:01.457476 19067 consensus_queue.cc:260] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.465258 19067 ts_tablet_manager.cc:1398] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240502 01:50:01.472776 18250 tablet_copy_service.cc:342] P 26c26643de084d6b8acaff73fa67f794: Request end of tablet copy session 542154990ef549669640e892eb3292e7-d5e09bfc9fdb46af9dd702b4c831e287 received from {username='slave'} at 127.0.0.1:46486
I20240502 01:50:01.473196 18250 tablet_copy_service.cc:433] P 26c26643de084d6b8acaff73fa67f794: ending tablet copy session 542154990ef549669640e892eb3292e7-d5e09bfc9fdb46af9dd702b4c831e287 on tablet d5e09bfc9fdb46af9dd702b4c831e287 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:01.473284 19054 ts_tablet_manager.cc:902] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:50:01.492645 19054 tablet_copy_client.cc:308] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.130:33629
I20240502 01:50:01.494690 18326 tablet_copy_service.cc:140] P 98f3bf374d49481cb6ebe0672e5b81dc: Received BeginTabletCopySession request for tablet 5090d855c6864a3d904fe1141d503d9d from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:40094)
I20240502 01:50:01.495211 18326 tablet_copy_service.cc:161] P 98f3bf374d49481cb6ebe0672e5b81dc: Beginning new tablet copy session on tablet 5090d855c6864a3d904fe1141d503d9d from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:40094: session id = 542154990ef549669640e892eb3292e7-5090d855c6864a3d904fe1141d503d9d
I20240502 01:50:01.501364 18326 tablet_copy_source_session.cc:205] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:01.506234 19054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5090d855c6864a3d904fe1141d503d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:01.519604 19054 tablet_copy_client.cc:783] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:01.520161 19054 tablet_copy_client.cc:647] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:01.523118 19054 tablet_copy_client.cc:523] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:01.530442 19054 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:01.547647 19054 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:01.548485 19054 tablet_bootstrap.cc:492] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:01.548982 19054 ts_tablet_manager.cc:1367] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.005s
I20240502 01:50:01.551093 19054 raft_consensus.cc:348] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.551885 19054 raft_consensus.cc:729] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:01.552386 19054 consensus_queue.cc:260] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:01.554610 19054 ts_tablet_manager.cc:1398] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:50:01.559387 18326 tablet_copy_service.cc:342] P 98f3bf374d49481cb6ebe0672e5b81dc: Request end of tablet copy session 542154990ef549669640e892eb3292e7-5090d855c6864a3d904fe1141d503d9d received from {username='slave'} at 127.0.0.1:40094
I20240502 01:50:01.559803 18326 tablet_copy_service.cc:433] P 98f3bf374d49481cb6ebe0672e5b81dc: ending tablet copy session 542154990ef549669640e892eb3292e7-5090d855c6864a3d904fe1141d503d9d on tablet 5090d855c6864a3d904fe1141d503d9d with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:01.642896 18992 raft_consensus.cc:1203] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:01.667668 18992 raft_consensus.cc:1203] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:01.748996 18992 raft_consensus.cc:1203] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:01.919672 18992 raft_consensus.cc:1203] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:02.134905 18960 raft_consensus.cc:1050] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:02.137077 18960 consensus_queue.cc:237] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.143571 18992 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.143815 18391 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.145260 19050 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.146787 19050 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.145709 18231 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.155683 19050 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:02.158593 19079 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.164865 19041 raft_consensus.cc:1050] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:02.167193 19041 consensus_queue.cc:237] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.168763 18992 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.173699 18991 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.175249 19022 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.175726 18392 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.192822 18391 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.194687 18891 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.197219 18129 catalog_manager.cc:5484] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.211808 18311 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.217922 19086 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:02.232250 18960 raft_consensus.cc:1050] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:02.234870 18960 consensus_queue.cc:237] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.241690 19022 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.243175 18392 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.248090 18992 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.250332 19079 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:02.231652 18231 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.256886 18991 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.265116 18237 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.267340 18960 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.268450 18309 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.269495 18388 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.273749 18960 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.299453 18129 catalog_manager.cc:5484] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.311657 19079 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.325501 18392 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.324651 18992 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.331025 18237 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.348476 18133 catalog_manager.cc:5484] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.356504 18309 consensus_queue.cc:237] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.363041 18314 consensus_queue.cc:237] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.363461 18992 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.364965 18960 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.367209 18237 consensus_queue.cc:237] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.372291 18234 raft_consensus.cc:1261] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.376209 18723 consensus_queue.cc:1035] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.380296 19050 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.381719 18991 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.383972 18991 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.385454 18392 raft_consensus.cc:1261] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.389859 18238 raft_consensus.cc:2932] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.391185 18118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3a4278616b44d738b2b7aa7eafef008 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:02.389884 18723 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240502 01:50:02.394608 18723 consensus_queue.cc:1035] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.400053 18392 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.401571 19041 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.412587 19041 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.417796 18992 raft_consensus.cc:1261] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.418668 18129 catalog_manager.cc:5484] T a3a4278616b44d738b2b7aa7eafef008 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.419418 18891 consensus_queue.cc:1035] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.420958 18388 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.431509 18119 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5e09bfc9fdb46af9dd702b4c831e287 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:02.435335 19079 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.440502 18990 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.450599 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a4278616b44d738b2b7aa7eafef008 with delete_type TABLET_DATA_TOMBSTONED (TS d6698cd7a2d740368c0c89233e16f04c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53772
I20240502 01:50:02.466094 18129 catalog_manager.cc:5484] T d5e09bfc9fdb46af9dd702b4c831e287 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:02.482065 19050 raft_consensus.cc:1050] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:02.484555 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e09bfc9fdb46af9dd702b4c831e287 with delete_type TABLET_DATA_TOMBSTONED (TS 98f3bf374d49481cb6ebe0672e5b81dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54726
I20240502 01:50:02.485299 19050 consensus_queue.cc:237] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.486542 18391 raft_consensus.cc:2932] T c3eec00736204e3fa0dcf0797f7b0d75 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.492396 18992 raft_consensus.cc:2932] T d5e09bfc9fdb46af9dd702b4c831e287 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.507103 18129 catalog_manager.cc:5484] T c3eec00736204e3fa0dcf0797f7b0d75 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.511104 19099 tablet_replica.cc:330] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: stopping tablet replica
I20240502 01:50:02.512033 18118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3eec00736204e3fa0dcf0797f7b0d75 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:02.535771 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet c3eec00736204e3fa0dcf0797f7b0d75 with delete_type TABLET_DATA_TOMBSTONED (TS 26c26643de084d6b8acaff73fa67f794 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41696
I20240502 01:50:02.541184 19100 tablet_replica.cc:330] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: stopping tablet replica
I20240502 01:50:02.542243 19100 raft_consensus.cc:2229] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:02.543009 19100 raft_consensus.cc:2258] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:02.547322 18388 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.524348 19099 raft_consensus.cc:2229] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:02.556692 19099 raft_consensus.cc:2258] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:02.557874 18992 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.560753 19079 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.561591 18238 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:02.562904 19079 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:02.580443 19079 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:02.599241 19099 ts_tablet_manager.cc:1875] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:02.609426 18960 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.607235 19100 ts_tablet_manager.cc:1875] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:02.614732 18238 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.620407 19103 tablet_replica.cc:330] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: stopping tablet replica
I20240502 01:50:02.621703 19103 raft_consensus.cc:2229] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:02.622606 19103 raft_consensus.cc:2258] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:02.630336 18388 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.641296 18131 catalog_manager.cc:5484] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:02.660609 18992 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.664402 19103 ts_tablet_manager.cc:1875] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:02.675580 19099 ts_tablet_manager.cc:1888] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:02.676153 19099 log.cc:1196] T a3a4278616b44d738b2b7aa7eafef008 P d6698cd7a2d740368c0c89233e16f04c: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root/wals/a3a4278616b44d738b2b7aa7eafef008
I20240502 01:50:02.678041 18119 catalog_manager.cc:4830] TS d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): tablet a3a4278616b44d738b2b7aa7eafef008 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:02.684387 18314 consensus_queue.cc:237] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:02.690402 19100 ts_tablet_manager.cc:1888] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:02.690898 19100 log.cc:1196] T d5e09bfc9fdb46af9dd702b4c831e287 P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root/wals/d5e09bfc9fdb46af9dd702b4c831e287
I20240502 01:50:02.698307 18992 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.700095 18118 catalog_manager.cc:4830] TS 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): tablet d5e09bfc9fdb46af9dd702b4c831e287 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:02.702175 18960 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.704263 18388 raft_consensus.cc:1261] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:02.705565 18960 consensus_queue.cc:1035] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:02.713500 19103 ts_tablet_manager.cc:1888] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:02.712741 19102 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.713953 19103 log.cc:1196] T c3eec00736204e3fa0dcf0797f7b0d75 P 26c26643de084d6b8acaff73fa67f794: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root/wals/c3eec00736204e3fa0dcf0797f7b0d75
I20240502 01:50:02.716581 18119 catalog_manager.cc:4830] TS 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): tablet c3eec00736204e3fa0dcf0797f7b0d75 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:02.715464 18388 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.718019 18992 raft_consensus.cc:2932] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.723171 18118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5090d855c6864a3d904fe1141d503d9d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:02.737709 18131 catalog_manager.cc:5484] T 5090d855c6864a3d904fe1141d503d9d P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:02.746974 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet 5090d855c6864a3d904fe1141d503d9d with delete_type TABLET_DATA_TOMBSTONED (TS 26c26643de084d6b8acaff73fa67f794 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41696
I20240502 01:50:02.747833 19103 tablet_replica.cc:330] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: stopping tablet replica
I20240502 01:50:02.748730 19103 raft_consensus.cc:2229] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:02.749357 19103 raft_consensus.cc:2258] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:02.752341 19103 ts_tablet_manager.cc:1875] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:02.766880 19103 ts_tablet_manager.cc:1888] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:02.767287 19103 log.cc:1196] T 5090d855c6864a3d904fe1141d503d9d P 26c26643de084d6b8acaff73fa67f794: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root/wals/5090d855c6864a3d904fe1141d503d9d
I20240502 01:50:02.772188 18119 catalog_manager.cc:4830] TS 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): tablet 5090d855c6864a3d904fe1141d503d9d (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:03.794553 18238 consensus_queue.cc:237] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
W20240502 01:50:03.802393 18204 consensus_peers.cc:485] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caecb89696f549ac8c4ab781e42472b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:03.803227 18314 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.803227 18391 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.804670 19086 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.805415 19041 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.813216 19041 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.825434 18311 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.831533 18388 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.836081 18231 consensus_queue.cc:237] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:03.843068 18129 catalog_manager.cc:5484] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:50:03.846213 18204 consensus_peers.cc:485] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92c0d6ed72f045a3b39631b83a054d78. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:03.846614 18391 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.850831 19086 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.859750 18311 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.861244 19111 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.863700 19112 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.867903 18392 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.874775 18311 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.896696 18132 catalog_manager.cc:5484] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.903296 18392 consensus_queue.cc:237] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:03.915232 18311 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.915836 18238 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.917218 19027 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.918104 19123 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:50:03.930030 18357 consensus_peers.cc:485] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ba72fe24abe4fe893e60d48e6679580. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:03.932827 19027 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.934774 18238 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.946336 18231 consensus_queue.cc:237] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:03.952872 18309 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:50:03.953835 18204 consensus_peers.cc:485] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9be322d2860f40408a8cd8199731477c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:03.954787 19112 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.955579 18129 catalog_manager.cc:5484] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:03.958820 18392 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:03.958089 18314 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.968168 19112 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:03.993659 19041 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:03.997606 18392 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:04.007383 18309 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:04.014695 18131 catalog_manager.cc:5484] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:04.343349 19132 ts_tablet_manager.cc:902] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775)
I20240502 01:50:04.355356 19135 ts_tablet_manager.cc:902] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:50:04.361910 19132 tablet_copy_client.cc:308] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.131:45775
I20240502 01:50:04.362274 19135 tablet_copy_client.cc:308] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.129:38913
I20240502 01:50:04.365005 18250 tablet_copy_service.cc:140] P 26c26643de084d6b8acaff73fa67f794: Received BeginTabletCopySession request for tablet 92c0d6ed72f045a3b39631b83a054d78 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:46486)
I20240502 01:50:04.365543 18250 tablet_copy_service.cc:161] P 26c26643de084d6b8acaff73fa67f794: Beginning new tablet copy session on tablet 92c0d6ed72f045a3b39631b83a054d78 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:46486: session id = 542154990ef549669640e892eb3292e7-92c0d6ed72f045a3b39631b83a054d78
I20240502 01:50:04.372016 18250 tablet_copy_source_session.cc:205] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:04.375257 19135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c0d6ed72f045a3b39631b83a054d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:04.380815 18402 tablet_copy_service.cc:140] P d6698cd7a2d740368c0c89233e16f04c: Received BeginTabletCopySession request for tablet 0ba72fe24abe4fe893e60d48e6679580 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:58642)
I20240502 01:50:04.381464 18402 tablet_copy_service.cc:161] P d6698cd7a2d740368c0c89233e16f04c: Beginning new tablet copy session on tablet 0ba72fe24abe4fe893e60d48e6679580 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:58642: session id = 542154990ef549669640e892eb3292e7-0ba72fe24abe4fe893e60d48e6679580
I20240502 01:50:04.389307 18402 tablet_copy_source_session.cc:205] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:04.394073 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba72fe24abe4fe893e60d48e6679580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:04.411541 19132 tablet_copy_client.cc:783] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:04.412395 19132 tablet_copy_client.cc:647] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:04.417268 19132 tablet_copy_client.cc:523] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:04.419090 19135 tablet_copy_client.cc:783] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:04.419703 19135 tablet_copy_client.cc:647] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:04.423305 19135 tablet_copy_client.cc:523] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:04.425030 19132 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:04.427906 19144 ts_tablet_manager.cc:902] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:50:04.431026 19135 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:04.452733 19132 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:04.453639 19132 tablet_bootstrap.cc:492] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:04.454344 19132 ts_tablet_manager.cc:1367] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.001s
I20240502 01:50:04.457254 19132 raft_consensus.cc:348] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.458170 19132 raft_consensus.cc:729] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:04.458837 19132 consensus_queue.cc:260] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.463456 19132 ts_tablet_manager.cc:1398] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240502 01:50:04.469738 19135 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:04.470820 19135 tablet_bootstrap.cc:492] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:04.471622 19135 ts_tablet_manager.cc:1367] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.016s
I20240502 01:50:04.474714 19135 raft_consensus.cc:348] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.475600 19135 raft_consensus.cc:729] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:04.477246 19135 consensus_queue.cc:260] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.479977 19135 ts_tablet_manager.cc:1398] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240502 01:50:04.488193 18250 tablet_copy_service.cc:342] P 26c26643de084d6b8acaff73fa67f794: Request end of tablet copy session 542154990ef549669640e892eb3292e7-92c0d6ed72f045a3b39631b83a054d78 received from {username='slave'} at 127.0.0.1:46486
I20240502 01:50:04.488677 18250 tablet_copy_service.cc:433] P 26c26643de084d6b8acaff73fa67f794: ending tablet copy session 542154990ef549669640e892eb3292e7-92c0d6ed72f045a3b39631b83a054d78 on tablet 92c0d6ed72f045a3b39631b83a054d78 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:04.489153 19144 tablet_copy_client.cc:308] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.129:38913
I20240502 01:50:04.492056 18250 tablet_copy_service.cc:140] P 26c26643de084d6b8acaff73fa67f794: Received BeginTabletCopySession request for tablet caecb89696f549ac8c4ab781e42472b1 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:46486)
I20240502 01:50:04.492615 18250 tablet_copy_service.cc:161] P 26c26643de084d6b8acaff73fa67f794: Beginning new tablet copy session on tablet caecb89696f549ac8c4ab781e42472b1 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:46486: session id = 542154990ef549669640e892eb3292e7-caecb89696f549ac8c4ab781e42472b1
I20240502 01:50:04.497223 18402 tablet_copy_service.cc:342] P d6698cd7a2d740368c0c89233e16f04c: Request end of tablet copy session 542154990ef549669640e892eb3292e7-0ba72fe24abe4fe893e60d48e6679580 received from {username='slave'} at 127.0.0.1:58642
I20240502 01:50:04.497702 18402 tablet_copy_service.cc:433] P d6698cd7a2d740368c0c89233e16f04c: ending tablet copy session 542154990ef549669640e892eb3292e7-0ba72fe24abe4fe893e60d48e6679580 on tablet 0ba72fe24abe4fe893e60d48e6679580 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:04.502936 18250 tablet_copy_source_session.cc:205] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:04.507174 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caecb89696f549ac8c4ab781e42472b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:04.525848 19144 tablet_copy_client.cc:783] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:04.526440 19144 tablet_copy_client.cc:647] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:04.530823 19144 tablet_copy_client.cc:523] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:04.535816 19132 ts_tablet_manager.cc:902] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:50:04.546824 19144 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:04.576402 19144 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:04.577334 19144 tablet_bootstrap.cc:492] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:04.578047 19144 ts_tablet_manager.cc:1367] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.015s
I20240502 01:50:04.580829 19144 raft_consensus.cc:348] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.581583 19144 raft_consensus.cc:729] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:04.582226 19144 consensus_queue.cc:260] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.587343 19144 ts_tablet_manager.cc:1398] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240502 01:50:04.587636 19132 tablet_copy_client.cc:308] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.129:38913
I20240502 01:50:04.589030 18250 tablet_copy_service.cc:342] P 26c26643de084d6b8acaff73fa67f794: Request end of tablet copy session 542154990ef549669640e892eb3292e7-caecb89696f549ac8c4ab781e42472b1 received from {username='slave'} at 127.0.0.1:46486
I20240502 01:50:04.589481 18250 tablet_copy_service.cc:433] P 26c26643de084d6b8acaff73fa67f794: ending tablet copy session 542154990ef549669640e892eb3292e7-caecb89696f549ac8c4ab781e42472b1 on tablet caecb89696f549ac8c4ab781e42472b1 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:04.590464 18249 tablet_copy_service.cc:140] P 26c26643de084d6b8acaff73fa67f794: Received BeginTabletCopySession request for tablet 9be322d2860f40408a8cd8199731477c from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:46486)
I20240502 01:50:04.591035 18249 tablet_copy_service.cc:161] P 26c26643de084d6b8acaff73fa67f794: Beginning new tablet copy session on tablet 9be322d2860f40408a8cd8199731477c from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:46486: session id = 542154990ef549669640e892eb3292e7-9be322d2860f40408a8cd8199731477c
I20240502 01:50:04.599077 18249 tablet_copy_source_session.cc:205] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:04.606230 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be322d2860f40408a8cd8199731477c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:04.620083 19132 tablet_copy_client.cc:783] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:04.620664 19132 tablet_copy_client.cc:647] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:04.623906 19132 tablet_copy_client.cc:523] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:04.632325 19132 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:04.675264 19132 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:04.676239 19132 tablet_bootstrap.cc:492] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:04.676877 19132 ts_tablet_manager.cc:1367] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.026s
I20240502 01:50:04.679162 19132 raft_consensus.cc:348] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.680045 19132 raft_consensus.cc:729] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:04.680658 19132 consensus_queue.cc:260] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:04.683048 19132 ts_tablet_manager.cc:1398] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:50:04.684983 18249 tablet_copy_service.cc:342] P 26c26643de084d6b8acaff73fa67f794: Request end of tablet copy session 542154990ef549669640e892eb3292e7-9be322d2860f40408a8cd8199731477c received from {username='slave'} at 127.0.0.1:46486
I20240502 01:50:04.685323 18249 tablet_copy_service.cc:433] P 26c26643de084d6b8acaff73fa67f794: ending tablet copy session 542154990ef549669640e892eb3292e7-9be322d2860f40408a8cd8199731477c on tablet 9be322d2860f40408a8cd8199731477c with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:04.906019 18992 raft_consensus.cc:1203] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:04.925801 18992 raft_consensus.cc:1203] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:04.925825 18990 raft_consensus.cc:1203] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:04.941167 18992 raft_consensus.cc:1203] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:05.341874 19041 raft_consensus.cc:1050] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:05.344295 19041 consensus_queue.cc:237] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.350296 18392 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.350517 18309 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.351027 18990 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.352209 19112 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.354064 19112 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.354898 19111 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.372467 19111 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.374246 18990 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.389343 18129 catalog_manager.cc:5484] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.397842 18309 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.401723 18392 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.430181 18231 consensus_queue.cc:237] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.436537 18990 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.437713 18314 raft_consensus.cc:1261] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.438289 19041 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.439249 19111 consensus_queue.cc:1035] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.446743 19161 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.448282 18314 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.448540 18990 raft_consensus.cc:2932] T 9be322d2860f40408a8cd8199731477c P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.456813 18119 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9be322d2860f40408a8cd8199731477c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:05.466178 18129 catalog_manager.cc:5484] T 9be322d2860f40408a8cd8199731477c P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.487039 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be322d2860f40408a8cd8199731477c with delete_type TABLET_DATA_TOMBSTONED (TS d6698cd7a2d740368c0c89233e16f04c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53772
I20240502 01:50:05.489216 19128 raft_consensus.cc:1050] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:05.489239 19169 tablet_replica.cc:330] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: stopping tablet replica
I20240502 01:50:05.490527 19169 raft_consensus.cc:2229] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:05.491043 19169 raft_consensus.cc:2258] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:05.491793 19128 consensus_queue.cc:237] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.494521 19169 ts_tablet_manager.cc:1875] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:05.498864 18990 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.498965 18309 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.500710 19158 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.501945 19128 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.503250 18231 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.505261 19158 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.516366 19169 ts_tablet_manager.cc:1888] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:05.516829 19169 log.cc:1196] T 9be322d2860f40408a8cd8199731477c P d6698cd7a2d740368c0c89233e16f04c: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root/wals/9be322d2860f40408a8cd8199731477c
I20240502 01:50:05.519093 18119 catalog_manager.cc:4830] TS d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): tablet 9be322d2860f40408a8cd8199731477c (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:05.520121 19161 raft_consensus.cc:1050] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:05.524447 19161 consensus_queue.cc:237] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.531394 19111 raft_consensus.cc:1050] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:05.535511 19111 consensus_queue.cc:237] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.550880 18992 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.551489 18392 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.552668 19112 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 01:50:05.554790 19177 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.547966 19128 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.549381 18990 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.561584 18314 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.563989 19111 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.578467 18310 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.579923 19112 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.578078 18309 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.578521 18991 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.588742 19111 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.588531 18129 catalog_manager.cc:5484] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.611697 18234 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.619135 18388 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:05.620682 19177 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:05.638156 19161 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.640552 18310 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.640312 18391 consensus_queue.cc:237] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.642944 19041 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.645347 18314 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.658501 18131 catalog_manager.cc:5484] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.659953 18991 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.659334 18387 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.661723 19171 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.665136 18992 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.672966 18234 raft_consensus.cc:1261] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer d6698cd7a2d740368c0c89233e16f04c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.672410 18991 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.674872 19158 consensus_queue.cc:1035] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.672483 19171 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P d6698cd7a2d740368c0c89233e16f04c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.686173 18129 catalog_manager.cc:5484] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:05.700570 18132 catalog_manager.cc:5484] T 0ba72fe24abe4fe893e60d48e6679580 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New cstate: current_term: 1 leader_uuid: "d6698cd7a2d740368c0c89233e16f04c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.709340 18119 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ba72fe24abe4fe893e60d48e6679580 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:05.718571 18231 raft_consensus.cc:2932] T 0ba72fe24abe4fe893e60d48e6679580 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.726490 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba72fe24abe4fe893e60d48e6679580 with delete_type TABLET_DATA_TOMBSTONED (TS 98f3bf374d49481cb6ebe0672e5b81dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54726
I20240502 01:50:05.715581 18391 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.735389 19188 tablet_replica.cc:330] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: stopping tablet replica
I20240502 01:50:05.739418 18237 consensus_queue.cc:237] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.716774 18992 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.757956 18991 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.758886 18310 raft_consensus.cc:1261] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.760078 19111 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.761111 19161 consensus_queue.cc:1035] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.770328 19177 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.775563 18234 consensus_queue.cc:237] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:05.776870 18314 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.779199 18991 raft_consensus.cc:2932] T 92c0d6ed72f045a3b39631b83a054d78 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.781164 18388 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.784448 19161 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:05.785945 18119 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92c0d6ed72f045a3b39631b83a054d78 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:05.796231 19188 raft_consensus.cc:2229] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:05.795289 19111 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.796963 19188 raft_consensus.cc:2258] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:05.796638 18387 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.806491 19188 ts_tablet_manager.cc:1875] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:05.813050 18131 catalog_manager.cc:5484] T 92c0d6ed72f045a3b39631b83a054d78 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:05.825139 18119 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet caecb89696f549ac8c4ab781e42472b1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:05.828720 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c0d6ed72f045a3b39631b83a054d78 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:53772
I20240502 01:50:05.829659 19169 tablet_replica.cc:330] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: stopping tablet replica
I20240502 01:50:05.830657 19169 raft_consensus.cc:2229] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:05.831490 19169 raft_consensus.cc:2258] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:05.830703 18990 raft_consensus.cc:1261] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:05.836239 19184 consensus_queue.cc:1035] T caecb89696f549ac8c4ab781e42472b1 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:50:05.840021 19169 ts_tablet_manager.cc:1875] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:05.845778 19188 ts_tablet_manager.cc:1888] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:05.844749 18990 raft_consensus.cc:2932] T caecb89696f549ac8c4ab781e42472b1 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.846374 19188 log.cc:1196] T 0ba72fe24abe4fe893e60d48e6679580 P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root/wals/0ba72fe24abe4fe893e60d48e6679580
I20240502 01:50:05.848058 18118 catalog_manager.cc:4830] TS 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): tablet 0ba72fe24abe4fe893e60d48e6679580 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:05.853061 18132 catalog_manager.cc:5484] T caecb89696f549ac8c4ab781e42472b1 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:05.859405 19169 ts_tablet_manager.cc:1888] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:05.859901 19169 log.cc:1196] T 92c0d6ed72f045a3b39631b83a054d78 P d6698cd7a2d740368c0c89233e16f04c: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root/wals/92c0d6ed72f045a3b39631b83a054d78
I20240502 01:50:05.861521 18119 catalog_manager.cc:4830] TS d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): tablet 92c0d6ed72f045a3b39631b83a054d78 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:05.863955 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet caecb89696f549ac8c4ab781e42472b1 with delete_type TABLET_DATA_TOMBSTONED (TS 98f3bf374d49481cb6ebe0672e5b81dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54726
I20240502 01:50:05.864811 19188 tablet_replica.cc:330] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: stopping tablet replica
I20240502 01:50:05.865796 19188 raft_consensus.cc:2229] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:05.866439 19188 raft_consensus.cc:2258] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:05.870208 19188 ts_tablet_manager.cc:1875] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:05.926200 19188 ts_tablet_manager.cc:1888] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:05.927379 19188 log.cc:1196] T caecb89696f549ac8c4ab781e42472b1 P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root/wals/caecb89696f549ac8c4ab781e42472b1
I20240502 01:50:05.930557 18118 catalog_manager.cc:4830] TS 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): tablet caecb89696f549ac8c4ab781e42472b1 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:06.926519 18314 consensus_queue.cc:237] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:06.933101 18234 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:06.934217 18387 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:50:06.933449 18281 consensus_peers.cc:485] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2318233df2844c48918882f56859c152. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:06.936129 19107 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:06.939021 19192 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:06.945274 19107 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:06.946990 18237 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:06.950660 18387 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:06.964121 18314 consensus_queue.cc:237] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:06.966646 18132 catalog_manager.cc:5484] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:50:07.015650 18281 consensus_peers.cc:485] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5411115a92043f684df0c01d3cc5e36. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:07.025753 18387 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:07.033195 19196 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:07.039140 18237 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:07.044611 19192 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:07.067667 19197 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.072660 18391 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.079478 18237 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.085201 18309 consensus_queue.cc:237] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.093256 18131 catalog_manager.cc:5484] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.101373 18391 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:50:07.102147 18281 consensus_peers.cc:485] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a92fc4beef0b4747bcf45cbabe4bb118. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:07.103346 19196 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:07.106491 18237 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:07.112753 19196 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:07.127127 19196 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.129419 18387 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.137359 18231 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.153076 18237 consensus_queue.cc:237] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.151173 18133 catalog_manager.cc:5484] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.170142 18387 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:50:07.171509 18204 consensus_peers.cc:485] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 810e60f2654045a39491fd7e358029da. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:07.171531 19112 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:07.171993 18314 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:07.173249 19177 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:50:07.185392 19111 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.186650 18309 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.212903 18131 catalog_manager.cc:5484] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:07.216370 18388 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:07.386224 19212 ts_tablet_manager.cc:902] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:50:07.397101 19212 tablet_copy_client.cc:308] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.130:33629
I20240502 01:50:07.399202 18326 tablet_copy_service.cc:140] P 98f3bf374d49481cb6ebe0672e5b81dc: Received BeginTabletCopySession request for tablet b5411115a92043f684df0c01d3cc5e36 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:40094)
I20240502 01:50:07.399892 18326 tablet_copy_service.cc:161] P 98f3bf374d49481cb6ebe0672e5b81dc: Beginning new tablet copy session on tablet b5411115a92043f684df0c01d3cc5e36 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:40094: session id = 542154990ef549669640e892eb3292e7-b5411115a92043f684df0c01d3cc5e36
I20240502 01:50:07.408064 18326 tablet_copy_source_session.cc:205] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:07.410998 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5411115a92043f684df0c01d3cc5e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:07.426591 19212 tablet_copy_client.cc:783] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:07.427173 19212 tablet_copy_client.cc:647] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:07.431557 19212 tablet_copy_client.cc:523] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:07.440235 19217 ts_tablet_manager.cc:902] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:50:07.441713 19212 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:07.458935 19217 tablet_copy_client.cc:308] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.130:33629
I20240502 01:50:07.460589 18326 tablet_copy_service.cc:140] P 98f3bf374d49481cb6ebe0672e5b81dc: Received BeginTabletCopySession request for tablet 2318233df2844c48918882f56859c152 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:40094)
I20240502 01:50:07.461135 18326 tablet_copy_service.cc:161] P 98f3bf374d49481cb6ebe0672e5b81dc: Beginning new tablet copy session on tablet 2318233df2844c48918882f56859c152 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:40094: session id = 542154990ef549669640e892eb3292e7-2318233df2844c48918882f56859c152
I20240502 01:50:07.465281 19212 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:07.466260 19212 tablet_bootstrap.cc:492] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:07.466981 19212 ts_tablet_manager.cc:1367] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.002s
I20240502 01:50:07.469187 18326 tablet_copy_source_session.cc:205] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:07.469954 19212 raft_consensus.cc:348] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.470788 19212 raft_consensus.cc:729] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:07.471307 19212 consensus_queue.cc:260] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.475270 19217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2318233df2844c48918882f56859c152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:07.478194 19212 ts_tablet_manager.cc:1398] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20240502 01:50:07.479853 18326 tablet_copy_service.cc:342] P 98f3bf374d49481cb6ebe0672e5b81dc: Request end of tablet copy session 542154990ef549669640e892eb3292e7-b5411115a92043f684df0c01d3cc5e36 received from {username='slave'} at 127.0.0.1:40094
I20240502 01:50:07.480378 18326 tablet_copy_service.cc:433] P 98f3bf374d49481cb6ebe0672e5b81dc: ending tablet copy session 542154990ef549669640e892eb3292e7-b5411115a92043f684df0c01d3cc5e36 on tablet b5411115a92043f684df0c01d3cc5e36 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:07.496927 19217 tablet_copy_client.cc:783] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:07.497514 19217 tablet_copy_client.cc:647] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:07.503657 19217 tablet_copy_client.cc:523] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:07.511616 19217 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:07.528731 19217 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:07.529526 19217 tablet_bootstrap.cc:492] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:07.530139 19217 ts_tablet_manager.cc:1367] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.011s
I20240502 01:50:07.532757 19217 raft_consensus.cc:348] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.533455 19217 raft_consensus.cc:729] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:07.534029 19217 consensus_queue.cc:260] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.536414 19217 ts_tablet_manager.cc:1398] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20240502 01:50:07.538539 18326 tablet_copy_service.cc:342] P 98f3bf374d49481cb6ebe0672e5b81dc: Request end of tablet copy session 542154990ef549669640e892eb3292e7-2318233df2844c48918882f56859c152 received from {username='slave'} at 127.0.0.1:40094
I20240502 01:50:07.538959 18326 tablet_copy_service.cc:433] P 98f3bf374d49481cb6ebe0672e5b81dc: ending tablet copy session 542154990ef549669640e892eb3292e7-2318233df2844c48918882f56859c152 on tablet 2318233df2844c48918882f56859c152 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:07.586741 19217 ts_tablet_manager.cc:902] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
I20240502 01:50:07.615345 19217 tablet_copy_client.cc:308] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.129:38913
I20240502 01:50:07.618451 18249 tablet_copy_service.cc:140] P 26c26643de084d6b8acaff73fa67f794: Received BeginTabletCopySession request for tablet 810e60f2654045a39491fd7e358029da from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:46486)
I20240502 01:50:07.619079 18249 tablet_copy_service.cc:161] P 26c26643de084d6b8acaff73fa67f794: Beginning new tablet copy session on tablet 810e60f2654045a39491fd7e358029da from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:46486: session id = 542154990ef549669640e892eb3292e7-810e60f2654045a39491fd7e358029da
I20240502 01:50:07.628053 18249 tablet_copy_source_session.cc:205] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:07.632470 19217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e60f2654045a39491fd7e358029da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:07.640254 19212 ts_tablet_manager.cc:902] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
I20240502 01:50:07.653693 19217 tablet_copy_client.cc:783] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:07.657047 19217 tablet_copy_client.cc:647] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:07.665407 19212 tablet_copy_client.cc:308] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.165.130:33629
I20240502 01:50:07.667241 18326 tablet_copy_service.cc:140] P 98f3bf374d49481cb6ebe0672e5b81dc: Received BeginTabletCopySession request for tablet a92fc4beef0b4747bcf45cbabe4bb118 from peer 542154990ef549669640e892eb3292e7 ({username='slave'} at 127.0.0.1:40094)
I20240502 01:50:07.667826 18326 tablet_copy_service.cc:161] P 98f3bf374d49481cb6ebe0672e5b81dc: Beginning new tablet copy session on tablet a92fc4beef0b4747bcf45cbabe4bb118 from peer 542154990ef549669640e892eb3292e7 at {username='slave'} at 127.0.0.1:40094: session id = 542154990ef549669640e892eb3292e7-a92fc4beef0b4747bcf45cbabe4bb118
I20240502 01:50:07.675716 18326 tablet_copy_source_session.cc:205] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:07.679642 19217 tablet_copy_client.cc:523] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:07.680644 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92fc4beef0b4747bcf45cbabe4bb118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:07.688694 19217 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:07.696413 19212 tablet_copy_client.cc:783] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:07.696972 19212 tablet_copy_client.cc:647] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:07.700448 19212 tablet_copy_client.cc:523] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:07.710558 19212 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: Bootstrap starting.
I20240502 01:50:07.714848 19217 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:07.715847 19217 tablet_bootstrap.cc:492] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:07.716428 19217 ts_tablet_manager.cc:1367] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.003s
I20240502 01:50:07.718858 19217 raft_consensus.cc:348] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.719658 19217 raft_consensus.cc:729] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:07.720376 19217 consensus_queue.cc:260] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.723037 19217 ts_tablet_manager.cc:1398] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:50:07.725194 18249 tablet_copy_service.cc:342] P 26c26643de084d6b8acaff73fa67f794: Request end of tablet copy session 542154990ef549669640e892eb3292e7-810e60f2654045a39491fd7e358029da received from {username='slave'} at 127.0.0.1:46486
I20240502 01:50:07.725510 18249 tablet_copy_service.cc:433] P 26c26643de084d6b8acaff73fa67f794: ending tablet copy session 542154990ef549669640e892eb3292e7-810e60f2654045a39491fd7e358029da on tablet 810e60f2654045a39491fd7e358029da with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:07.736582 19212 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:07.737393 19212 tablet_bootstrap.cc:492] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: Bootstrap complete.
I20240502 01:50:07.738086 19212 ts_tablet_manager.cc:1367] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.004s
I20240502 01:50:07.742623 19212 raft_consensus.cc:348] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.743312 19212 raft_consensus.cc:729] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 542154990ef549669640e892eb3292e7, State: Initialized, Role: LEARNER
I20240502 01:50:07.743976 19212 consensus_queue.cc:260] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:07.746119 19212 ts_tablet_manager.cc:1398] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:50:07.747974 18326 tablet_copy_service.cc:342] P 98f3bf374d49481cb6ebe0672e5b81dc: Request end of tablet copy session 542154990ef549669640e892eb3292e7-a92fc4beef0b4747bcf45cbabe4bb118 received from {username='slave'} at 127.0.0.1:40094
I20240502 01:50:07.748375 18326 tablet_copy_service.cc:433] P 98f3bf374d49481cb6ebe0672e5b81dc: ending tablet copy session 542154990ef549669640e892eb3292e7-a92fc4beef0b4747bcf45cbabe4bb118 on tablet a92fc4beef0b4747bcf45cbabe4bb118 with peer 542154990ef549669640e892eb3292e7
I20240502 01:50:07.983170 18990 raft_consensus.cc:1203] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:07.993477 18990 raft_consensus.cc:1203] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:08.152906 18990 raft_consensus.cc:1203] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:08.186570 18990 raft_consensus.cc:1203] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:08.449769 19237 raft_consensus.cc:1050] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:08.452287 19237 consensus_queue.cc:237] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.459221 18990 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.460880 19196 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.461793 18237 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.463236 18391 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.464270 19196 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.465184 19237 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.474500 19196 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.481500 18990 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.484490 18234 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.486303 18387 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.494886 18132 catalog_manager.cc:5484] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:08.504995 19196 raft_consensus.cc:1050] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:08.508456 19196 consensus_queue.cc:237] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.525715 18990 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.529456 19102 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:08.532801 18391 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.532800 18231 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.536113 19237 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.542354 19236 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:08.571843 18992 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.569711 19237 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.579560 18237 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.586092 18314 consensus_queue.cc:237] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.589488 18391 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.597844 19238 raft_consensus.cc:1050] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:08.600400 18133 catalog_manager.cc:5484] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.600960 19238 consensus_queue.cc:237] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.614935 18991 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:08.616823 19196 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:08.620446 18387 raft_consensus.cc:1261] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:08.632733 19248 consensus_queue.cc:1035] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:08.640718 18992 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.642606 18990 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.639175 19237 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.651782 18314 consensus_queue.cc:237] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.658457 19235 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.662827 18388 raft_consensus.cc:2932] T 2318233df2844c48918882f56859c152 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.665421 18314 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.665787 18991 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:08.663972 18391 raft_consensus.cc:1261] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:08.679270 18118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2318233df2844c48918882f56859c152 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:08.681993 19196 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:50:08.681928 18131 catalog_manager.cc:5484] T 2318233df2844c48918882f56859c152 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.697513 19238 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:08.698767 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet 2318233df2844c48918882f56859c152 with delete_type TABLET_DATA_TOMBSTONED (TS 26c26643de084d6b8acaff73fa67f794 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41696
I20240502 01:50:08.700804 18388 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.715365 19246 consensus_queue.cc:1035] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:50:08.716723 19257 tablet_replica.cc:330] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: stopping tablet replica
I20240502 01:50:08.718016 19177 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.720453 19257 raft_consensus.cc:2229] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:08.722218 19257 raft_consensus.cc:2258] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:08.725548 19248 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.732896 18992 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.733393 19177 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.734666 18990 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.741297 18387 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.745932 19257 ts_tablet_manager.cc:1875] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:08.749588 18118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5411115a92043f684df0c01d3cc5e36 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:08.749150 18131 catalog_manager.cc:5484] T b5411115a92043f684df0c01d3cc5e36 P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.751660 18391 raft_consensus.cc:2932] T b5411115a92043f684df0c01d3cc5e36 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26c26643de084d6b8acaff73fa67f794 (127.16.165.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.779170 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet b5411115a92043f684df0c01d3cc5e36 with delete_type TABLET_DATA_TOMBSTONED (TS 26c26643de084d6b8acaff73fa67f794 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41696
I20240502 01:50:08.780931 19237 raft_consensus.cc:1050] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc: attempting to promote NON_VOTER 542154990ef549669640e892eb3292e7 to VOTER
I20240502 01:50:08.784335 18131 catalog_manager.cc:5484] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.784683 18314 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.790169 19237 consensus_queue.cc:237] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.823300 18231 consensus_queue.cc:237] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:08.827791 18992 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 LEARNER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.828344 18234 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.828583 18387 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:08.831442 19196 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.834095 18387 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:08.834308 19237 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:08.852123 18990 raft_consensus.cc:1261] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:08.853848 19257 ts_tablet_manager.cc:1888] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:08.854573 19177 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:08.859527 19246 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:08.860481 19257 log.cc:1196] T 2318233df2844c48918882f56859c152 P 26c26643de084d6b8acaff73fa67f794: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root/wals/2318233df2844c48918882f56859c152
I20240502 01:50:08.865506 19177 consensus_queue.cc:1035] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:08.868721 18119 catalog_manager.cc:4830] TS 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): tablet 2318233df2844c48918882f56859c152 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:08.869155 19257 tablet_replica.cc:330] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: stopping tablet replica
I20240502 01:50:08.870541 19257 raft_consensus.cc:2229] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:08.873188 19257 raft_consensus.cc:2258] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:08.884255 19235 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.890231 18992 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.907181 18391 raft_consensus.cc:2932] T 810e60f2654045a39491fd7e358029da P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.908244 19257 ts_tablet_manager.cc:1875] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:08.912992 18131 catalog_manager.cc:5484] T 810e60f2654045a39491fd7e358029da P 542154990ef549669640e892eb3292e7 reported cstate change: config changed from index 3 to 4, VOTER 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130) evicted. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.921676 18119 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 810e60f2654045a39491fd7e358029da with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:08.924367 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet 810e60f2654045a39491fd7e358029da with delete_type TABLET_DATA_TOMBSTONED (TS 98f3bf374d49481cb6ebe0672e5b81dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54726
I20240502 01:50:08.934438 19266 tablet_replica.cc:330] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: stopping tablet replica
I20240502 01:50:08.945525 19266 raft_consensus.cc:2229] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:08.946503 19266 raft_consensus.cc:2258] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:08.959169 19266 ts_tablet_manager.cc:1875] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:08.965811 19257 ts_tablet_manager.cc:1888] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:08.976174 19257 log.cc:1196] T b5411115a92043f684df0c01d3cc5e36 P 26c26643de084d6b8acaff73fa67f794: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-0-root/wals/b5411115a92043f684df0c01d3cc5e36
I20240502 01:50:08.973145 19247 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.980878 18119 catalog_manager.cc:4830] TS 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913): tablet b5411115a92043f684df0c01d3cc5e36 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:08.981130 18387 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.986569 18231 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.990372 19266 ts_tablet_manager.cc:1888] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:08.990849 19266 log.cc:1196] T 810e60f2654045a39491fd7e358029da P 98f3bf374d49481cb6ebe0672e5b81dc: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-1-root/wals/810e60f2654045a39491fd7e358029da
I20240502 01:50:08.991674 18992 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:08.995496 18118 catalog_manager.cc:4830] TS 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629): tablet 810e60f2654045a39491fd7e358029da (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:09.008998 18129 catalog_manager.cc:5484] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index 2 to 3, 542154990ef549669640e892eb3292e7 (127.16.165.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:09.027004 18314 consensus_queue.cc:237] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } }
I20240502 01:50:09.035682 18237 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:09.035715 18992 raft_consensus.cc:1261] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:09.036923 19196 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:09.039343 19246 consensus_queue.cc:1035] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:09.045236 19196 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:09.047586 18237 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:09.048852 18992 raft_consensus.cc:2932] T a92fc4beef0b4747bcf45cbabe4bb118 P 542154990ef549669640e892eb3292e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:09.062721 18118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a92fc4beef0b4747bcf45cbabe4bb118 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:09.062119 18131 catalog_manager.cc:5484] T a92fc4beef0b4747bcf45cbabe4bb118 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index 3 to 4, VOTER d6698cd7a2d740368c0c89233e16f04c (127.16.165.131) evicted. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: false } } }
I20240502 01:50:09.094333 18373 tablet_service.cc:1514] Processing DeleteTablet for tablet a92fc4beef0b4747bcf45cbabe4bb118 with delete_type TABLET_DATA_TOMBSTONED (TS d6698cd7a2d740368c0c89233e16f04c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53772
I20240502 01:50:09.105912 19267 tablet_replica.cc:330] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: stopping tablet replica
I20240502 01:50:09.107056 19267 raft_consensus.cc:2229] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:09.107744 19267 raft_consensus.cc:2258] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:09.110095 19267 ts_tablet_manager.cc:1875] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:09.121035 19267 ts_tablet_manager.cc:1888] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:09.121356 19267 log.cc:1196] T a92fc4beef0b4747bcf45cbabe4bb118 P d6698cd7a2d740368c0c89233e16f04c: Deleting WAL directory at /tmp/dist-test-taskpdFmK8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714614525285352-17046-0/minicluster-data/ts-2-root/wals/a92fc4beef0b4747bcf45cbabe4bb118
I20240502 01:50:09.122742 18119 catalog_manager.cc:4830] TS d6698cd7a2d740368c0c89233e16f04c (127.16.165.131:45775): tablet a92fc4beef0b4747bcf45cbabe4bb118 (table test-workload [id=530c909a97e04e548fa7ff60e9c29bb3]) successfully deleted
I20240502 01:50:10.133828 18314 consensus_queue.cc:237] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:10.139267 18387 raft_consensus.cc:1261] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.139685 18237 raft_consensus.cc:1261] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Refusing update from remote peer 98f3bf374d49481cb6ebe0672e5b81dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:50:10.140100 18281 consensus_peers.cc:485] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd5d4f10e3a24f8e854fda3cbd4b0a0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:10.140668 19269 consensus_queue.cc:1035] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:10.141624 19196 consensus_queue.cc:1035] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:10.160245 19269 raft_consensus.cc:2932] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.174252 18237 raft_consensus.cc:2932] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 26c26643de084d6b8acaff73fa67f794 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.186689 18391 raft_consensus.cc:2932] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.188983 18131 catalog_manager.cc:5484] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 98f3bf374d49481cb6ebe0672e5b81dc reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:10.189893 18231 consensus_queue.cc:237] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:10.203179 18387 raft_consensus.cc:1261] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.204334 19235 consensus_queue.cc:1035] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:10.204499 18314 raft_consensus.cc:1261] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.206943 19235 consensus_queue.cc:1035] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:50:10.208308 18204 consensus_peers.cc:485] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae19377bfe6e48e481f6054c06e669b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:10.215382 19211 raft_consensus.cc:2932] T ae19377bfe6e48e481f6054c06e669b6 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.219430 18388 raft_consensus.cc:2932] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.243695 18231 consensus_queue.cc:237] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
I20240502 01:50:10.249459 18314 raft_consensus.cc:2932] T ae19377bfe6e48e481f6054c06e669b6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
W20240502 01:50:10.251430 18204 consensus_peers.cc:485] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42d668f31e144969aae9c1410dd481e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:10.256121 18387 raft_consensus.cc:1261] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.259665 18310 raft_consensus.cc:1261] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.258587 19278 consensus_queue.cc:1035] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240502 01:50:10.262722 19211 consensus_queue.cc:1035] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:10.279667 18388 raft_consensus.cc:2932] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.277899 19282 raft_consensus.cc:2932] T 42d668f31e144969aae9c1410dd481e6 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.294454 18132 catalog_manager.cc:5484] T ae19377bfe6e48e481f6054c06e669b6 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.304104 18310 raft_consensus.cc:2932] T 42d668f31e144969aae9c1410dd481e6 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.314365 18133 catalog_manager.cc:5484] T 42d668f31e144969aae9c1410dd481e6 P d6698cd7a2d740368c0c89233e16f04c reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } attrs { replace: true } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.324991 18231 consensus_queue.cc:237] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } }
W20240502 01:50:10.335770 18204 consensus_peers.cc:485] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 -> Peer 542154990ef549669640e892eb3292e7 (127.16.165.132:45743): Couldn't send request to peer 542154990ef549669640e892eb3292e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b45ab88cf544fca9ca712cfaa2c601. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:10.337160 18388 raft_consensus.cc:1261] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.339197 19278 consensus_queue.cc:1035] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:10.342067 18314 raft_consensus.cc:1261] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Refusing update from remote peer 26c26643de084d6b8acaff73fa67f794: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:10.348117 19278 consensus_queue.cc:1035] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:10.355371 19282 raft_consensus.cc:2932] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.356565 18387 raft_consensus.cc:2932] T c3b45ab88cf544fca9ca712cfaa2c601 P d6698cd7a2d740368c0c89233e16f04c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.378593 18133 catalog_manager.cc:5484] T c3b45ab88cf544fca9ca712cfaa2c601 P 26c26643de084d6b8acaff73fa67f794 reported cstate change: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New cstate: current_term: 1 leader_uuid: "26c26643de084d6b8acaff73fa67f794" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:10.389497 18310 raft_consensus.cc:2932] T c3b45ab88cf544fca9ca712cfaa2c601 P 98f3bf374d49481cb6ebe0672e5b81dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 542154990ef549669640e892eb3292e7 (127.16.165.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "26c26643de084d6b8acaff73fa67f794" member_type: VOTER last_known_addr { host: "127.16.165.129" port: 38913 } } peers { permanent_uuid: "98f3bf374d49481cb6ebe0672e5b81dc" member_type: VOTER last_known_addr { host: "127.16.165.130" port: 33629 } } peers { permanent_uuid: "d6698cd7a2d740368c0c89233e16f04c" member_type: VOTER last_known_addr { host: "127.16.165.131" port: 45775 } attrs { replace: true } } peers { permanent_uuid: "542154990ef549669640e892eb3292e7" member_type: NON_VOTER last_known_addr { host: "127.16.165.132" port: 45743 } attrs { promote: true } } }
I20240502 01:50:10.615137 19292 ts_tablet_manager.cc:902] T ae19377bfe6e48e481f6054c06e669b6 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
F20240502 01:50:10.616155 19292 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0ba72fe24abe4fe893e60d48e6679580.queue_time_us, tablet-download-pool-0ba72fe24abe4fe893e60d48e6679580.queue_time_us) (tablet-download-pool-0ba72fe24abe4fe893e60d48e6679580.run_cpu_time_us, tablet-download-pool-0ba72fe24abe4fe893e60d48e6679580.run_cpu_time_us) (tablet-download-pool-0ba72fe24abe4fe893e60d48e6679580.run_wall_time_us, tablet-download-pool-0ba72fe24abe4fe893e60d48e6679580.run_wall_time_us) (tablet-download-pool-2318233df2844c48918882f56859c152.queue_time_us, tablet-download-pool-2318233df2844c48918882f56859c152.queue_time_us) (tablet-download-pool-2318233df2844c48918882f56859c152.run_cpu_time_us, tablet-download-pool-2318233df2844c48918882f56859c152.run_cpu_time_us) (tablet-download-pool-2318233df2844c48918882f56859c152.run_wall_time_us, tablet-download-pool-2318233df2844c48918882f56859c152.run_wall_time_us) (tablet-download-pool-5090d855c6864a3d904fe1141d503d9d.queue_time_us, tablet-download-pool-5090d855c6864a3d904fe1141d503d9d.queue_time_us) (tablet-download-pool-5090d855c6864a3d904fe1141d503d9d.run_cpu_time_us, tablet-download-pool-5090d855c6864a3d904fe1141d503d9d.run_cpu_time_us) (tablet-download-pool-5090d855c6864a3d904fe1141d503d9d.run_wall_time_us, tablet-download-pool-5090d855c6864a3d904fe1141d503d9d.run_wall_time_us) (tablet-download-pool-810e60f2654045a39491fd7e358029da.queue_time_us, tablet-download-pool-810e60f2654045a39491fd7e358029da.queue_time_us) (tablet-download-pool-810e60f2654045a39491fd7e358029da.run_cpu_time_us, tablet-download-pool-810e60f2654045a39491fd7e358029da.run_cpu_time_us) (tablet-download-pool-810e60f2654045a39491fd7e358029da.run_wall_time_us, tablet-download-pool-810e60f2654045a39491fd7e358029da.run_wall_time_us) (tablet-download-pool-92c0d6ed72f045a3b39631b83a054d78.queue_time_us, tablet-download-pool-92c0d6ed72f045a3b39631b83a054d78.queue_time_us) (tablet-download-pool-92c0d6ed72f045a3b39631b83a054d78.run_cpu_time_us, tablet-download-pool-92c0d6ed72f045a3b39631b83a054d78.run_cpu_time_us) (tablet-download-pool-92c0d6ed72f045a3b39631b83a054d78.run_wall_time_us, tablet-download-pool-92c0d6ed72f045a3b39631b83a054d78.run_wall_time_us) (tablet-download-pool-9be322d2860f40408a8cd8199731477c.queue_time_us, tablet-download-pool-9be322d2860f40408a8cd8199731477c.queue_time_us) (tablet-download-pool-9be322d2860f40408a8cd8199731477c.run_cpu_time_us, tablet-download-pool-9be322d2860f40408a8cd8199731477c.run_cpu_time_us) (tablet-download-pool-9be322d2860f40408a8cd8199731477c.run_wall_time_us, tablet-download-pool-9be322d2860f40408a8cd8199731477c.run_wall_time_us) (tablet-download-pool-a3a4278616b44d738b2b7aa7eafef008.queue_time_us, tablet-download-pool-a3a4278616b44d738b2b7aa7eafef008.queue_time_us) (tablet-download-pool-a3a4278616b44d738b2b7aa7eafef008.run_cpu_time_us, tablet-download-pool-a3a4278616b44d738b2b7aa7eafef008.run_cpu_time_us) (tablet-download-pool-a3a4278616b44d738b2b7aa7eafef008.run_wall_time_us, tablet-download-pool-a3a4278616b44d738b2b7aa7eafef008.run_wall_time_us) (tablet-download-pool-a92fc4beef0b4747bcf45cbabe4bb118.queue_time_us, tablet-download-pool-a92fc4beef0b4747bcf45cbabe4bb118.queue_time_us) (tablet-download-pool-a92fc4beef0b4747bcf45cbabe4bb118.run_cpu_time_us, tablet-download-pool-a92fc4beef0b4747bcf45cbabe4bb118.run_cpu_time_us) (tablet-download-pool-a92fc4beef0b4747bcf45cbabe4bb118.run_wall_time_us, tablet-download-pool-a92fc4beef0b4747bcf45cbabe4bb118.run_wall_time_us) (tablet-download-pool-ae19377bfe6e48e481f6054c06e669b6.queue_time_us, tablet-download-pool-ae19377bfe6e48e481f6054c06e669b6.queue_time_us) (tablet-download-pool-b5411115a92043f684df0c01d3cc5e36.queue_time_us, tablet-download-pool-b5411115a92043f684df0c01d3cc5e36.queue_time_us) (tablet-download-pool-b5411115a92043f684df0c01d3cc5e36.run_cpu_time_us, tablet-download-pool-b5411115a92043f684df0c01d3cc5e36.run_cpu_time_us) (tablet-download-pool-b5411115a92043f684df0c01d3cc5e36.run_wall_time_us, tablet-download-pool-b5411115a92043f684df0c01d3cc5e36.run_wall_time_us) (tablet-download-pool-c3eec00736204e3fa0dcf0797f7b0d75.queue_time_us, tablet-download-pool-c3eec00736204e3fa0dcf0797f7b0d75.queue_time_us) (tablet-download-pool-c3eec00736204e3fa0dcf0797f7b0d75.run_cpu_time_us, tablet-download-pool-c3eec00736204e3fa0dcf0797f7b0d75.run_cpu_time_us) (tablet-download-pool-c3eec00736204e3fa0dcf0797f7b0d75.run_wall_time_us, tablet-download-pool-c3eec00736204e3fa0dcf0797f7b0d75.run_wall_time_us) (tablet-download-pool-caecb89696f549ac8c4ab781e42472b1.queue_time_us, tablet-download-pool-caecb89696f549ac8c4ab781e42472b1.queue_time_us) (tablet-download-pool-caecb89696f549ac8c4ab781e42472b1.run_cpu_time_us, tablet-download-pool-caecb89696f549ac8c4ab781e42472b1.run_cpu_time_us) (tablet-download-pool-caecb89696f549ac8c4ab781e42472b1.run_wall_time_us, tablet-download-pool-caecb89696f549ac8c4ab781e42472b1.run_wall_time_us) (tablet-download-pool-d5e09bfc9fdb46af9dd702b4c831e287.queue_time_us, tablet-download-pool-d5e09bfc9fdb46af9dd702b4c831e287.queue_time_us) (tablet-download-pool-d5e09bfc9fdb46af9dd702b4c831e287.run_cpu_time_us, tablet-download-pool-d5e09bfc9fdb46af9dd702b4c831e287.run_cpu_time_us) (tablet-download-pool-d5e09bfc9fdb46af9dd702b4c831e287.run_wall_time_us, tablet-download-pool-d5e09bfc9fdb46af9dd702b4c831e287.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714614610 (unix time) try "date -d @1714614610" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004296) received by PID 17046 (TID 0x7fd907bfd700) from PID 17046; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fd9d53d7980 (unknown) at ??:0
I20240502 01:50:10.696705 19294 ts_tablet_manager.cc:902] T fd5d4f10e3a24f8e854fda3cbd4b0a0d P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 98f3bf374d49481cb6ebe0672e5b81dc (127.16.165.130:33629)
@ 0x7fd9ce94bfb7 gsignal at ??:0
@ 0x7fd9ce94d921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fd9d1741da4 google::LogMessage::SendToLog() at ??:0
I20240502 01:50:10.748275 19296 ts_tablet_manager.cc:902] T c3b45ab88cf544fca9ca712cfaa2c601 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
@ 0x7fd9d1742950 google::LogMessage::Flush() at ??:0
@ 0x7fd9d1747a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:50:10.788713 19297 ts_tablet_manager.cc:902] T 42d668f31e144969aae9c1410dd481e6 P 542154990ef549669640e892eb3292e7: Initiating tablet copy from peer 26c26643de084d6b8acaff73fa67f794 (127.16.165.129:38913)
@ 0x7fd9d20cf3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd9d20a594f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fd9d20a30a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fd9d9ec3422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fd9d9ec9c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fd9d9f5f4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fd9d9f903cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fd9d9f761ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fd9d9f7617a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7fd9d9f7610a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fd9d9f760d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fd9d9f752fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fd9db31bda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fd9db31bbda std::__1::function<>::operator()() at ??:0
@ 0x7fd9d20a7237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd9d20aa692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd9d20aa64a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7fd9d20aa5da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fd9d20aa5a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fd9d20a989e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fd9db31bda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fd9db31bbda std::__1::function<>::operator()() at ??:0
@ 0x7fd9d2089937 kudu::Thread::SuperviseThread() at ??:0